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1.0 


SUMMARY 


Everyone  agrees  that  intelligence  analysis  is  an  intensely  cognitive  process  and  that  the  quality  of 
an  analyst’s  cognition  determines  the  quality  of  his  or  her  analysis.  However,  less  is  understood 
about  what  elements  combine  to  make  productive,  effective  cognition.  The  objective  of  the 
Advanced  Analysis  Cognition  project  is  to  improve  human  performance  during  the 
analysis/synthesis  process  by  analyzing  the  results  of  research  in  several  aspects  of  human 
cognition  to  understand  what  contributes  to  effective  cognitive  performance. 

This  report  on  thinking  dispositions  is  the  second  in  a  series  of  reports  on  different  aspects  of 
analytic  cognition.  My  primary  research  questions  were: 

What  is  the  value  and  feasibility  of  evaluating  a  candidate’s  thinking  dispositions  as  one 
component  of  intelligence  analyst  selection?  What  is  the  feasibility  of  training  individuals 
in  order  to  strengthen  their  positive  thinking  dispositions? 

I  found: 

•  Scholars  in  multiple  disciplines  share  a  common  understanding  of  thinking 
dispositions.  Considerable  overlap  exists  among  the  specific  thinking  dispositions 
identified  in  the  published  research. 

•  Thinking  dispositions  do  have  an  impact  on  cognitive  performance.  Individuals 
with  similar  Intelligence  Quotients  (IQs)  perform  differently  on  cognitive  tasks 
depending  on  the  strength  of  their  dispositions  toward  different  aspects  of  cognitive 
effort. 

•  An  individual’s  thinking  disposition  strength  can  be  measured. 

•  A  strong  overlap  exists  between  thinking  dispositions  found  for  this  report  and  what 
scholars  in  the  intelligence  community  have  identified  as  characteristics  of  effective 
intelligence  analysts. 

•  Individuals  can  strengthen  their  positive  thinking  dispositions  with  focused  effort. 

Based  on  these  findings,  I  believe  it  is  practical  and  valuable  to  assess  a  potential  intelligence 
analyst’s  thinking  dispositions  as  one  component  of  analyst  selection.  In  addition,  currently 
employed  intelligence  analysts  can  improve  their  positive  thinking  dispositions  through  targeted 
performance  improvement  events. 
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2.0 


INTRODUCTION 


Everyone  agrees  that  intelligence  analysis  is  an  intensely  cognitive  process  and  that  the  quality  of 
an  analyst’s  cognition  determines  the  quality  of  his  or  her  analysis.  However,  less  is  understood 
about  what  elements  combine  to  make  productive,  effective  cognition. 

Richards  Heuer  noted  this  challenge  in  Psychology  of  Intelligence  Analysis : 

Of  the  diverse  problems  that  impede  accurate  intelligence  analysis,  those  inherent  in 
human  mental  processes  are  surely  among  the  most  important  and  most  difficult  to  deal 
with.  Intelligence  analysis  is  fundamentally  a  mental  process,  but  understanding  this 
process  is  hindered  by  the  lack  of  conscious  awareness  of  the  workings  of  our  own 
minds.1 2 3 

The  objective  of  the  Advanced  Analysis  Cognition  project  is  to  improve  human  performance 
during  the  analysis/synthesis  process.  To  accomplish  this  we  have  analyzed  the  results  of 
research  in  several  areas  of  human  cognition  in  order  to  understand  what  contributes  to  effective 
cognitive  performance.  We  hope  to  increase  conscious  awareness  about  the  workings  of  our 
minds.  This  report  on  thinking  dispositions  is  the  second  in  a  series  of  reports  on  different 
aspects  of  analytic  cognition. 

Thinking  dispositions  are  a  theoretical  construct  intended  to  describe  one  aspect  of  cognition. 
They  describe  a  person’s  inclination  or  motivation  to  make  an  effort  to  use  one’s  mental  skills  to 
address  tasks  requiring  thinking. 

The  Framework  for  Analytic  Cognition  (FAC):  A  Guide  for  Doing  All-Source  Intelligence 
Analysis  noted  the  contribution  that  thinking  dispositions  make  to  effective  cognition: 

While  there  are  many  unknowns  about  how  people  think  and  various  terms  are  used,  the 
preceding  discussion  indicates  the  following  factors  are  important. 

•  Knowledge 

•  Thinking  dispositions 

•  Cognitive  skills 

•  Motivation  and  engagement 

2 

•  Epistemological  beliefs 

Advanced  Analysis  Cognition:  Critical  Thinking  also  highlighted  the  importance  of  thinking 
dispositions  to  effective  thought: 

The  ability  to  employ  critical  thinking  skills  is  still  not  sufficient  to  ensure  that  an 
individual  will  bring  a  critical  approach  to  thinking.  A  thinker  must  be  willing  to  make 
the  extra  effort  to  think  critically.  The  thinker  must  have  the  disposition  to  think 

■j 

critically. 

The  earliest  scholars  interested  in  improving  thinking  highlighted  the  importance  of  attitude.  For 
example,  John  Dewey  noted  that  a  thinker  needs  more  than  skills  in  How  We  Think. 


1  Heuer,  R.J.J.  Psychology  of  Intelligence  Analysis,  Center  for  the  Study  of  Intelligence,  Washington,  D.C.,  1999,  p. 
1. 

2  Kampman,  C.M.,  Mangio,  C.A.,  Parry,  T.L.  &  Wilkinson,  B.J.  Framework  for  Analytic  Cognition  (FAC):  A  Guide 
for  Doing  all-Source  Intelligence  Analysis,  in  press. 

3  Marsh,  M.  Advanced  Analytic  Cognition:  Critical  Thinking,  in  press. 
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Knowledge  of  the  methods  alone  will  not  suffice;  there  must  be  the  desire,  the  will,  to 
employ  them.  This  desire  is  an  affair  of  personal  disposition.  But  on  the  other  hand  the 
disposition  alone  will  not  suffice.  There  must  also  be  understanding  of  the  forms  and 
techniques  that  are  the  channels  through  which  these  attitudes  operate  to  the  best 
advantage.4 

Edward  Glaser  described  the  importance  of  thinking  dispositions  in  his  1938  experiment 
regarding  teaching  critical  thinking: 

The  development  of  ability  to  think  critically,  it  should  be  noted,  is  not  limited  to 
cultivation  of  better  methods  for  finding  and  testing  evidence  and  meanings,  and  arriving 
at  well-founded  conclusions.  Knowledge  of  the  methods  of  logical  inquiry  is  important. 
Even  more  important  for  the  everyday  practice  of  democracy,  however,  are  the  attitudes 
involved  in  critical  thinking.  Persons  who  have  acquired  a  disposition  to  want  evidence 
for  beliefs,  and  who  have  acquired  an  attitude  of  reasonableness  have  also  acquired 
something  of  a  way  of  life  which  makes  for  more  considerate  and  humane  relationships 
among  men.5 

Our  understanding  of  the  contribution  of  thinking  dispositions  toward  effective  thinking  led  us  to 
explore  further  what  has  been  researched  about  thinking  dispositions.  We  hoped  to  determine  if 
improving  positive  thinking  dispositions  could  benefit  the  intelligence  analysis  process. 

2.1  Purpose 

My  primary  research  questions  are: 

What  is  the  value  and  feasibility  of  evaluating  a  candidate’s  thinking  dispositions  as  one 
component  of  intelligence  analyst  selection?  What  is  the  feasibility  of  training 
individuals  in  order  to  strengthen  their  positive  thinking  dispositions? 

To  answer  this  question,  I  explored  the  following  topics: 

•  What  are  thinking  dispositions? 

o  How  are  they  described?  Do  different  professional  disciplines  have 
different  perspectives  on  thinking  dispositions? 

o  How  are  they  related  to  various  descriptions  of  thinking  such  as  critical 
thinking,  reflective  thinking,  or  rational  thinking? 

o  Are  related  concepts  that  describe  the  affective  or  motivational  aspects  of 
human  cognition  present  in  their  literature? 

o  How  have  thinking  dispositions  been  understood  over  time? 

•  How  do  thinking  dispositions  affect  cognitive  performance? 

o  How  do  individuals  use  their  thinking  dispositions  while  performing 
cognitive  tasks? 

o  Does  the  use  and  influence  of  thinking  dispositions  vary  depending  on  the 
nature  of  the  cognitive  task? 


4  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  p.  30. 

5  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p.  6. 
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Can  thinking  dispositions  be  assessed?  Do  reliable  and  valid  assessment  methods 
exist?  What  evidence  exists  that  supports  the  value  of  thinking  disposition 
assessments? 

How  do  thinking  dispositions  relate  to  the  characteristics,  attributes,  and  traits  of 
intelligence  analysts? 

How  are  an  individual's  thinking  dispositions  established  or  subsequently  changed? 
o  What  factors  influence  the  development  of  various  thinking  dispositions? 

o  Can  thinking  dispositions  be  improved  with  training  and  education  or  other 
factors  such  as  life  experience  or  maturity?  What  evidence  exists  for  this 
improvement? 
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3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 

This  project  builds  upon  work  completed  to  support  the  Framework  for  Analytic  Cognition6 
development  and  the  Advanced  Analysis  Cognition:  Critical  Thinking  research.7  As  a  result,  the 
initial  documents  searched  were  those  collected  for  these  two  projects.8 

3.1  Information  Acquisition 

The  steps  I  used  to  identify  potentially  relevant  documents  were: 

1 .  Search  all  documents  in  our  document  collection  for  the  tenn  thinking  dispositions. 

2.  Search  our  collection  for  documents  describing  analyst  attributes  or  characteristics. 

3.  For  those  documents  identified  as  containing  thinking  disposition  or  analyst 
attribute  information,  search  the  individual  document  to  identify  what  was  said. 

4.  Review  reference  lists  in  these  documents  to  identify  additional  documents  to 
review  and  evaluate. 

5.  Use  Google  search  to  identify  additional  documents,  articles,  and  resources.  Search 
terms  included:  thinking  dispositions,  cognitive  styles,  other  synonyms  for  thinking 
dispositions  found  in  the  documents,  author  names,  article  and  document  names 
found  in  references  within  documents,  and  tools  or  resources  mentioned  in 
documents. 

6.  Follow  interesting  leads  to  relevant  journals,  web  sites,  new  documents,  new 
authors,  and  new  tools. 

Using  these  steps,  I  reviewed  a  total  of  301  documents  that  appeared  to  be  potentially  relevant  to 
the  research  topics.  Research  and  reflection  about  thinking  dispositions  encompasses  multiple 
fields  of  study,  with  the  majority  of  documents  from  the  fields  of  philosophy,  psychology,  and 
education.  The  documents  addressed  thinking  disposition  concepts  and  theories,  as  well  as 
information  on  assessments,  training,  guidelines,  application,  and  the  results  of  experiments. 

The  collection  includes  documents  from  the  best  known  theorists  in  thinking  dispositions  (such 
as  Facione,  Tishman,  Perkins,  Stanovich,  and  Cacioppo)  as  well  as  from  scholars  who  were 
interested  in  a  specific  application  of  or  experiment  into  thinking  dispositions. 

3.2  This  Report 

This  report  evaluates  and  summarizes  the  results  of  the  research,  with  an  emphasis  on  how 
thinking  dispositions  might  be  used  in  selecting  intelligence  analysts  and  developing  their 
professional  performance. 

I  selected  publications  for  inclusion  in  the  report  to  illustrate  the  range  of  perspectives  present  in 
the  literature  of  thinking  dispositions.  I  also  selected  documents  that  highlight  areas  of 
convergence  that  address  the  primary  purpose  of  the  report  and  that  are  most  beneficial  for 
intelligence  analysis. 

The  remainder  of  the  report  is  organized  as  follows: 

4.0  Results  and  discussion 


6  Kampman  et  al.,  in  press. 

7  Marsh,  in  press. 

s  This  document  collection  currently  includes  over  5000  references  in  a  variety  of  fields  including  intelligence 
analysis,  cognitive  psychology,  philosophy,  training  and  education,  and  more. 
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4.0  RESULTS  AND  DISCUSSION 
4.1  What  Are  Thinking  Dispositions? 

In  this  section  of  the  report  I  explore  the  following  concepts  surrounding  thinking  dispositions: 

•  How  do  scholars  define  thinking  dispositions?  Do  different  professional  disciplines 
have  perspectives  on  thinking  dispositions?  Are  different  thinking  dispositions 
required  for  different  types  of  thinking  such  as  critical  thinking  reflective  thinking, 
or  rational  thinking? 

•  What  are  some  examples  of  thinking  dispositions? 

•  Are  there  related  concepts  describing  affective  or  motivational  aspects  of  cognition? 

•  How  have  thinking  dispositions  been  understood  over  time? 

The  goal  of  this  section  of  the  report  is  to  determine  if  sufficient  consistency  exists  in  the 
understanding  of  thinking  dispositions  that  it  enables  us  to  use  the  concept  for  selecting  and 
developing  intelligence  analysts. 

4.1.1.  Scholarly  Definitions  of  Thinking  Dispositions 

Thinking  dispositions  are  a  theoretical  construct  intended  to  describe  one  aspect  of  cognition  - 
the  inclination  or  motivation  to  make  an  effort  to  use  one’s  mental  skills  to  address  tasks 
requiring  thinking.  Although  scholars  in  the  fields  of  education,  philosophy  and  psychology 
conducted  most  of  the  thinking  dispositions  research,  researchers  in  other  fields  have  also 
investigated  the  topic.  I  found  that  the  conceptualization  of  thinking  dispositions  is  relatively 
consistent  between  scholars  and  across  fields.  I  have  included  three  examples  below  to  illustrate 
these  similarities. 

Educators  Tishman,  Perkins,  and  Jay  have  written  extensively  about  thinking  dispositions  and 
the  importance  of  attending  to  thinking  dispositions  when  teaching.  They  defined  thinking 
dispositions  for  good  thinking  as: 

Yet  what  sets  good  thinkers  apart  is  not  simply  superior  cognitive  ability  or  particular 
skills;  rather,  it  is  their  abiding  tendencies  to  explore,  inquire,  seek  clarity,  take 
intellectual  risks,  and  think  critically  and  imaginatively.  These  tendencies  can  be  called 
"thinking  dispositions."9 

Tishman,  Perkins  and  Jay  expanded  on  their  concept  of  thinking  dispositions  by  adding  the  idea 
of  sensitivity  as  part  of  thinking  dispositions  in  addition  to  inclination.  While  their  third 
component,  ability,  crosses  over  into  the  skills  area,  the  distinction  between  sensitivity  and 
inclination  is  an  important  one.  Someone  who  is  willing  to  put  effort  into  thinking  but  does  not 
recognize  when  this  effort  is  appropriate  may  still  not  engage  cognitively  when  needed. 

Specifically,  a  disposition  in  our  sense  is  a  psychological  element  with  three  components: 
inclination,  sensitivity,  and  ability.  The  three  components  contribute  as  follows: 
Inclination  refers  to  the  person's  felt  tendency  toward  behavior  X.  For  example,  a  person 
with  an  inclination  to  open-minded  thinking  will  feel  a  leaning  toward  open-minded 
thinking  when  he  or  she  discerns  the  need.  Sensitivity,  in  contrast,  refers  to  the  person’s 
alertness  to  X  occasions.  For  example,  a  person  sensitive  to  the  need  for  open-minded 
thinking  will  notice  occasions  where  narrow  thinking  and  prejudice  and  bias  are  likely 

9  Tishman,  S.,  Jay,  E.  &  Perkins,  D.N.  "Teaching  Thinking  Dispositions:  From  Transmission  to  Enculturation," 
Theory  into  Practice,  Vol.  32,  No.  3,  1993,  pp.  147-148. 
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and  open-mindedness  called  for.  Finally,  ability,  of  course,  refers  to  the  actual  ability  to 
follow  through  with  X  behavior.  A  person  with  the  ability  to  be  open-minded  knows  how 
to  go  about  it:  resisting  the  impulse  to  decide  quickly,  listening  to  evidence  for  rival 
points  of  view,  and  so  on.10 

Psychologist  Deanna  Kuhn  defined  the  dispositions  needed  to  support  critical  thinking  as: 

Critical  thinking  is  more  than  the  successful  use  of  a  particular  skill  in  an  appropriate 
context.  It  is  also  an  attitude  or  disposition  to  recognize  when  a  skill  is  needed  and  the 
willingness  to  apply  it.  Sears  and  Parsons  (1991)  called  these  dispositions  the  ethic  of  a 
critical  thinker.* 11 

Philosopher  Harvey  Siegel  described  the  dispositions  supporting  critical  thinking  as  providing 
someone  with  a  critical  spirit. 

In  order  to  be  a  critical  thinker,  a  person  must  have,  in  addition  to  what  has  been  said  thus 
far,  certain  attitudes,  dispositions,  habits  of  mind,  and  character  traits,  which  together 
may  be  labeled  the  "critical  attitude"  or  "critical  spirit."  Most  generally,  a  critical  thinker 
must  not  only  be  able  to  assess  reasons  properly,  in  accordance  with  the  reason 
assessment  component,  she  must  be  disposed  to  do  so  as  well;  that  is,  a  critical  thinker 
must  have  a  well-developed  disposition  to  engage  in  reason  assessment.  A  critical 
thinker  must  have  a  willingness  to  conform  judgment  and  action  to  principle,  not  simply 
an  ability  to  so  conform.12 

Although  the  vast  majority  of  scholars  whose  work  I  reviewed  for  this  study  believe  that  thinking 
dispositions  are  essential  components  of  effective  cognition,  I  did  find  one  contrary  view. 
Michael  Fisher  and  Alec  Scriven  noted: 

Now  this  distinction  between  critical  thinking  and  the  critical  thinker  may  seem  fairly 
obvious,  but  it's  a  distinction  that  has  been  treated  carelessly  by  several  writers  who  too 
quickly  argue  that  one  can't  do  critical  thinking  if  one  doesn't  have  the  critical  thinker's 
characteristically  active  approach.  Some  have  even  argued  that  you  can't  be  said  to  be 
doing  critical  thinking  unless  you  have  a  reliable  disposition  that  will  always  incline  you 
to  do  it.  Here,  we're  saying  that  one  can  often  do  critical  thinking  without  reaching  the 

higher  levels  of  activity  (sub-levels  3B  and  3C),  and  do  it  just  as  well  as  the 

1  "2 

characteristically  active  critical  thinker  would  do  it. 

A  common  understanding  of  the  definition  of  thinking  dispositions  is  clear  from  the  research: 
thinking  dispositions  represent  an  individual’s  inclination  toward  thinking.  Positive  or 
productive  thinking  dispositions  describe  the  tendency  to  notice  when  effortful  thinking  is 
required  and  the  inclination  to  make  the  effort  when  thinking  is  required.  Less  productive 
dispositions  might  lead  an  individual  toward  ineffective  approaches  to  thinking  tasks.  As 
Tishman,  Perkins,  and  Jay  noted: 


10  Perkins,  D.N.,  Jay,  E.  &  Tishman,  S.  "Beyond  Abilities:  A  Dispositional  Theory  of  Thinking,"  The  Merrill- 
P aimer  Quarterly,  Vol.  39,  No.  1,  1993,  p.  4. 

11  Halpern,  D.F.  "Teaching  Critical  Thinking  for  Transfer  Across  Domains  ”  American  Psychologist,  Vol.  53,  No.  4, 
1998,  p.  152. 

12  Siegel,  H.  Educating  Reason:  Rationality,  Critical  Thinking,  and  Education,  Routledge,  New  York,  NY,  1988,  p. 
39. 

13  Fisher,  A.  &  Scriven,  M.  Critical  Thinking:  Its  Definition  and  Assessment,  Edgepress,  Point  Reyes,  CA,  p.  46. 
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They  can  be  good  or  bad,  productive  or  counterproductive.  For  example,  you  might  have 
the  disposition  to  make  careful  plans  in  appropriate  situations.  That's  good.  But 
sometimes  you  might  have  the  disposition  to  plunge  ahead  blindly,  without  taking  the 
time  to  plan  or  think  ahead.  That’s  bad.14 

Someone  who  has  strong  positive  thinking  dispositions  and  weak  negative  thinking  dispositions 
is  more  likely  to  be  an  effective  thinker.  The  converse  is  also  true:  someone  who  has  weak 
positive  thinking  dispositions  or  strong  negative  thinking  dispositions  is  less  likely  to  be  an 
effective  thinker.  Section  4.3  of  this  report  addresses  how  we  can  assess  the  relative  strength  or 
weakness  of  an  individual’s  thinking  dispositions. 

With  a  clear  understanding  of  the  concept  of  thinking  dispositions,  let  us  evaluate  some  of  the 
specific  thinking  dispositions  identified  by  scholars  and  practitioners. 

4,1.2.  Examples  of  Thinking  Dispositions 

Because  thinking  dispositions  are  a  theoretical  construct,  moving  from  an  agreement  about  what 
thinking  dispositions  are  conceptually  to  specific  examples  of  useful  thinking  dispositions 
represents  a  minor  challenge.  Each  scholar  had  a  slightly  different  perspective  on  which 
thinking  dispositions  are  most  crucial  for  good  thinking.  For  example,  some  scholars  focused  on 
positive  thinking  dispositions  such  as  open-mindedness  while  others  focused  on  thinking 
dispositions  that  inhibit  effective  thinking  such  as  dogmatism. 

Even  when  the  concepts  expressed  by  these  examples  of  specific  thinking  dispositions  are 
similar,  the  language  used  to  describe  the  thinking  disposition  varied.  For  example,  Ennis’ 
disposition  to  seek  reasons ,  Tishman  et  al.  and  Zechmeister  &  Johnson’s  disposition  to  be 
intellectually  curious,  Stanovich’s  disposition  to  have  a  tendency  to  collect  information,  and 
Facione’s  disposition  of  truthseeking  all  appear  to  describe  the  same  inclination  toward  gathering 
information  before  coming  to  a  decision  or  conclusion. 

I  found  two  fundamentally  different  approaches  to  identifying  specific  thinking  dispositions: 

1 .  Scholars  who  addressed  the  question  of  what  makes  an  effective  thinker.  These 
scholars  tended  to  define  a  list  of  multiple  dispositions  in  support  of  some 
conception  of  thinking  such  as  good  thinking,  critical  thinking,  or  rational  thinking. 

2.  Scholars  who  addressed  the  question  of  what  causes  individual  differences  in 
cognitive  effort  and  effectiveness.  These  scholars  tended  to  focus  on  exploring  a 
single  disposition  and  measuring  its  impact  on  cognition. 

Dispositions  that  Characterize  an  Effective  Thinker.  The  lists  of  examples  included  in  this 
report  have  been  developed  by  scholars  in  different  fields  and  in  support  of  different  types  of 
thinking.  Despite  different  terminology,  strong  similarities  do  exist  between  the  lists.  For 
example,  the  majority  of  lists  addressed  the  disposition  of  being  open-minded  in  some  way. 

These  lists  also  recognize  that  the  inclination  to  avoid  biases  and  to  hold  decisions  or  conclusions 
until  appropriate  information  or  evidence  is  available  is  an  important  contributor  to  all  forms  of 
effective  thinking.  Another  very  commonly-identified  disposition  is  the  inclination  to  search  for 
relevant  information. 

I  found  documents  defining  and  describing  thinking  dispositions  to  support  a  wide  variety  of 
kinds  of  thinking.  Despite  the  different  types  of  thinking  the  dispositions  support,  I  found  no 


14  Tishman,  S.,  Jay,  E.  &  Perkins,  D.N.  "Teaching  Thinking  Dispositions:  From  Transmission  to  Enculturation," 
Theory  into  Practice,  Vol.  32,  No.  3,  1993,  p.  148. 
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significant  difference  between  dispositions  in  support  of  the  different  effective  thinking  types, 
including: 

•  Good  thinking 

•  Intelligent  thinking 

•  Rational  thinking 

•  Reflective  thinking 

•  Constructive  thinking 

•  Critical  thinking 

Table  1  consists  of  six  lists  of  dispositions  describing  some  kind  of  effective  thinking.15  The 
authors  featured  in  Table  1  and  their  backgrounds  include: 

•  Robert  Ennis  is  a  philosopher  and  educator  who  has  studied  critical  thinking 
extensively.  His  list  of  dispositions  was  published  in  his  article,  “A  Taxonomy  of 
Critical  Thinking  Dispositions  and  Abilities.” 

•  Educators  Tishman,  Perkins,  and  Jay  included  their  list  of  dispositions  supporting 
good  thinking  in  multiple  articles. 

•  Psychologists  Zechmeister  and  Johnson  included  a  description  of  critical  thinking 
dispositions  in  their  textbook  called  Critical  Thinking:  a  Functional  Approach. 

•  Psychologist  Keith  Stanovich  has  written  extensively  on  the  impact  that  thinking 
dispositions  have  on  reasoning  ability.  He  summarized  dispositions  that  lead  to 
rational  thinking  in  What  Intelligence  Tests  Miss:  the  Psychology  of  Rational 
Thought. 

•  The  organization  that  is  known  today  as  the  Foundation  for  Critical  Thinking  was 
established  in  1980  as  the  Center  for  Critical  Thinking.  The  Center  is  now  part  of 
the  Critical  Thinking  Community16  consisting  of  the  Foundation  for  Critical 
Thinking,  the  Center  for  Critical  Thinking,  the  National  Council  for  Excellence  in 
Critical  Thinking,  and  the  International  Center  for  the  Assessment  of  Higher  Order 
Thinking.  The  Foundation  published  a  booklet  entitled  Critical  Thinking 
Competency  Standards  which  included  a  list  of  dispositions. 

•  Philosopher  Peter  Facione  led  an  effort  for  the  American  Philosophical  Association 
(APA)  to  develop  a  consistent  understanding  of  critical  thinking.  This  effort 
resulted  in  what  has  been  called  the  consensus  definition  of  critical  thinking.  It  also 
provided  a  list  of  characteristics  describing  the  disposition  of  effective  critical 
thinkers.  Facione  later  developed  the  California  Critical  Thinking  Dispositions 
Inventory  (CCTDI)  to  assess  an  individual’s  disposition  to  think  critically  based  on 
the  results  of  the  APA  study.  In  the  course  of  the  development  of  the  CCTDI,  the 
long  list  of  characteristics  was  collapsed  into  a  shorter  list  of  broad  categories  of 
dispositions. 


15  Appendix  A  contains  all  thinking  dispositions  found  in  the  research  for  this  report. 

16  www.criticalthinking.org 
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Table  1:  Comparison  of  Thinking  Dispositions  by  Different  Scholars  for  Different  Types  of  Thinking 


Critical  Thinking  Dispositions  Defined  by 
Ennis  17 

1.  Seek  a  clear  statement  of  the  thesis  or  question 

2.  Seek  reasons 

3.  Try  to  be  well  informed 

4.  Use  and  mention  credible  sources 

5.  Take  into  account  the  total  situation 

6.  Try  to  remain  relevant  to  the  main  point 

7.  Keep  in  mind  the  original  and/or  basic  concern 

8.  Look  for  alternatives 

9.  Be  open-minded 

a)  Consider  seriously  other  points  of  view  than 
one's  own  (dialogical  thinking) 

b)  Reason  from  premises  with  which  one 
disagrees — without  letting  the  disagreement 
interfere  with  one's  reasoning  (suppositional 
thinking) 

c)  Withhold  judgment  when  the  evidence  and 
reasons  are  insufficient 

10.  Take  a  position  (and  change  a  position)  when 
the  evidence  and  reasons  are  sufficient  to  do  so 

11.  Seek  as  much  precision  as  the  subject  permits 

12.  Deal  in  an  orderly  manner  with  the  parts  of  a 
complex  whole 

13.  Use  one's  critical  thinking  abilities 

14.  Be  sensitive  to  the  feelings,  level  of 


Good  Thinkins  Dispositions  Defined  bv 

18 

Tishman,  Jay,  and  Perkins 

1.  The  disposition  to  be  broad  and  adventurous',  the 
tendency  to  be  open-minded,  to  explore  alternative 
views;  an  alertness  to  narrow  thinking;  the  ability 
to  generate  multiple  options. 

2.  The  disposition  toward  sustained  intellectual 
curiosity :  the  tendency  to  wonder,  probe,  find 
problems;  a  zest  for  inquiry;  an  alertness  for 
anomalies;  the  ability  to  observe  closely  and 
formulate  questions. 

3.  The  disposition  to  clarify  and  seek 
understanding',  a  desire  to  understand  clearly,  to 
seek  connections  and  explanations;  an  alertness  to 
unclarity  and  need  for  focus;  an  ability  to  build 
cone  eptualizations . 

4.  The  disposition  to  be  planful  and  strategic,  the 
drive  to  set  goals,  make  and  execute  plans,  envision 
outcomes;  alertness  to  lack  of  direction;  the  ability 
to  formulate  goals  and  plans. 

5.  The  disposition  to  be  intellectually  careful :  the 
urge  for  precision,  organization,  thoroughness;  an 
alertness  to  possible  error  or  inaccuracy;  the  ability 
to  process  information  precisely. 

6.  The  disposition  to  seek  and  evaluate  reasons :  the 
tendency  to  question  the  given,  to  demand 


Critical  Thinking  Dispositions  Defined  by 
Zechmeister  &  Johnson19 

1.  Intellectual  curiosity :  seeking  answers  to 
various  kinds  of  questions  and  problems. 

2.  Objectivity',  using  objective  factors  when  one 
makes  decisions  and  avoiding  being  influenced 
by  emotional  or  subjective  factors. 

3.  Open-mindedness',  being  willing  to  consider  a 
wide  variety  of  beliefs. 

4.  Flexibility,  exhibiting  a  willingness  to  change 
one's  beliefs  or  methods  of  inquiry. 

5.  Intellectual  skepticism',  not  accepting  a 
conclusion  as  being  true  until  adequate  evidence 
is  provided. 

6.  Intellectual  honesty,  accepting  statements  as 
true  even  when  they  don't  agree  with  one's  own 
position. 

7.  Being  systematic,  following  a  line  of  reasoning 
consistently  to  a  conclusion. 

8.  Persistence',  being  persistent  in  seeking  to 
resolve  disputes. 

9.  Decisiveness :  reaching  a  conclusion  when  the 
evidence  warrants  it. 

1 0.  Respect  for  other  viewpoints',  willingness  to 
admit  that  you  are  wrong  and  that  others  may  be 


17  Ennis,  R.H.  "A  Taxonomy  of  Critical  Thinking  Dispositions  and  Abilities,"  in  Teaching  Thinking  Skills :  Theory  and  Practice,  eds.  J.B.  Baron  &  R.J.  Sternberg, 
Freeman,  New  York,  NY,  1987,  p.  12. 

18  Tishman,  S.,  Jay,  E.  &  Perkins,  D.N.  "Teaching  Thinking  Dispositions:  From  Transmission  to  Enculturation,"  Theory  into  Practice,  Vol.  32,  No.  3,  1993,  p.  148. 

19  Zechmeister,  E.B.  &  Johnson,  J.E.  Critical  Thinking:  A  Functional  Approach,  Brooks/Cole  Publishing  Co.,  Pacific  Grove,  CA,  1992,  pp.  6-7. 
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knowledge,  and  degree  of  sophistication  of  others 

justification;  an  alertness  to  the  need  for  evidence; 
the  ability  to  weigh  and  assess  reasons. 

7.  The  disposition  to  be  metacognitive :  the 
tendency  to  be  aware  of  and  monitor  the  flow  of 
one's  own  thinking;  alertness  to  complex  thinking 
situations;  the  ability  to  exercise  control  of  mental 
processes  and  be  reflective. 

right. 

Rational  Thinking  Dispositions  defined  bv 
Stanovich  20 

the  tendency  to  collect  information  before 
making  up  one's  mind, 

the  tendency  to  seek  various  points  of  view 
before  coming  to  a  conclusion, 

the  disposition  to  think  extensively  about  a 
problem  before  responding, 

the  tendency  to  calibrate  the  degree  or  strength  of 
one's  opinion  to  the  degree  of  evidence  available, 

the  tendency  to  think  about  future  consequences 
before  taking  action, 

the  tendency  to  explicitly  weigh  pluses  and 
minuses  of  situations  before  making  a  decision, 
and 

the  tendency  to  seek  nuance  and  avoid 
absolutism. 

Critical  Thinking  Dispositions  defined  bv  the 
Foundation  for  Critical  Thinking21 

The  competencies  focusing  on  the  intellectual 
traits,  virtues  or  dispositions: 

Fairmindedness  -  seek  to  treat  all  viewpoints  with 
equality 

Intellectual  Humility  -  distinguish  what  they  know 
from  what  they  don't  know 

Intellectual  Courage  -  willing  to  challenge  popular 
beliefs 

Intellectual  Empathy  -  to  sympathetically  enter  into 
points  of  view  that  differ  from  their  own 

Intellectual  Integrity  -  hold  themselves  to  the  same 
standards  they  expect  others  to  meet 

Intellectual  Perseverance  -  work  through 
complexities  and  frustration  without  giving  up 

Confidence  in  Reason  -  recognize  that  good 
reasoning  is  the  key  to  living  a  rational  life,  and  to 
creating  a  more  fair  and  just  world 

Intellectual  Autonomy  -  take  responsibility  for  their 
own  thinking,  beliefs,  and  value 

Critical  thinking  Dispositions  defined  bv 
Facione22 

Truth-seeking  is  the  habit  of  always  desiring  the 
best  possible  understanding  of  any  given 
situation;  it  is  following  reasons  and  evidence 
wherever  they  may  lead,  even  if  they  lead  one  to 
question  cherished  beliefs. 

Open-mindedness  is  the  tendency  to  allow  others 
to  voice  views  with  which  one  may  not  agree. 
Analyticity  is  the  tendency  to  be  alert  to  what 
happens  next. 

Systematicity  is  the  tendency  or  habit  of  striving 
to  approach  problems  in  a  disciplined,  orderly, 
and  systematic  way. 

Critical  thinking  self-confidence  is  the  tendency 
to  trust  the  use  of  reason  and  reflective  thinking 
to  solve  problems. 

Inquisitiveness  is  intellectual  curiosity. 

Cognitive  maturity  (maturity  of  judgment)  is  the 
tendency  to  see  problems  as  complex,  rather  than 
black  and  white. 

20  Stanovich,  K.E.  What  Intelligence  Tests  Miss:  The  Psychology’  of  Rational  Thought,  Yale  University  Press,  New  Haven,  CT,  2009,  pp.  31-32. 

21  Paul,  R.  &  Elder,  L.  A  Guide  for  Educators  to  Critical  Thinking  Competency  Standards,  Foundation  for  Critical  Thinking,  Dillon  Beach,  CA,  2006,  pp.  26-32. 

22  Facione,  P.A.  &  Facione,  N.C.  California  Critical  Thinking  Disposition  Inventory  (CCTDI)  Test  Manual,  California  Academic  Press,  Millbrae,  CA,  2010,  pp.  9- 


10. 
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Table  2  includes  sample  dispositions  developed  for  specific  industries  and  organizations  as  part 
of  their  investigation  of  critical  thinking.  Again,  similarities  exist  between  these  and  the 
scholarly  lists  of  thinking  dispositions  as  shown  in  Table  1. 

•  Ed  Krupat  with  the  Center  for  Evaluation  Medical  Education  Grand  Rounds 
conducted  a  survey  of  practicing  Medical  Doctor  (MD)  faculty  at  five  medical 
schools.  Ten  percent  of  the  responses  described  critical  thinking  in  dispositional 
terms. 

•  Sharon  Staib  reported  on  a  study  in  the  nursing  profession  that  was  modeled  on 
Facione’s  project  for  the  APA.  This  consensus  definition  for  critical  thinking  in 
nursing  also  addressed  thinking  dispositions. 

•  The  planning  group  on  critical  thinking  at  Frostburg  State  University  developed  an 
operational  definition  of  critical  thinking  to  enable  the  university  to  teach  critical 
thinking  and  to  assess  the  ability  of  students  to  think  critically.  Their  definition 
included  thinking  dispositions. 


Table  2:  Comparison  of  Thinking  Dispositions  for  Specific  Industries  and  Organizations 


Critical  Thinking 

Critical  Thinking  Dispositions  for 

Critical  Thinking  Dispositions 

Dispositions  for  Physicians 

Nurses  Reported  by  Staib24 

Defined  by  Frostburg  State 

Summarized  by  Krupat23 

Critical  thinkers  in  nursing  exhibit 

University25 

•  . .  .careful  attention  to  what 

these  habits  of  the  mind: 

A  student  who  has  developed 

you  know,  vigilance  for  what 

•  confidence, 

critical  thinking  dispositions  will 

you  do  not,  and  the  courage  to 

show  the  inclination  to 

question  both  of  the  above 

•  contextual  perspective, 

•  approach  questions  with 
an  open-minded  and 

categories. 

•  creativity, 

•  . .  .thinking  about  an  topic, 

•  flexibility, 

curious  attitude,  be 

issue,  or  challenge  in  a  way 

•  inquisitiveness, 

informed  by  multiple 

that  sets  aside  my  immediate 

•  intellectual  integrity, 

relevant  perspectives,  and 

’gut’  response,  so  that  I  can  be 

be  willing  to  examine 

open  and  reflective  to  other 

•  intuition, 

questions  in  a  fair-minded 

possible  ways  of  viewing  the 

•  open-mindedness, 

way; 

challenge... 

•  perseverance,  and 

•  apply  critical  thinking 

•  . .  .thinking  deeply,  keenly, 
flexibly,  openly,  reflectively, 
with  an  awareness  of  self  and 
others,  with  attention  to  what 
is  known  and  unknown,  and 
with  humility. 

•  reflection 

skills  to  thinking  about 
issues  in  a  major  field  of 
study,  general  education 
courses,  and  everyday 
lives; 

•  reflect  on  how  best  to 
answer  questions,  solve 
problems,  and  make 
decisions  in  academic  and 
everyday  settings. 

23  Krupat,  E.  Critical  Thinking  in  Medical  Education:  Assessing  What  We  Mean  and  What  We  Know  [Powerpoint 
Presentation] ,  2008,  p.  14. 

24  Staib,  S.  "Teaching  and  Measuring  Critical  Thinking,"  Journal  of  Nursing  Education,  Vol.  42,  No.  11,  2003,  p. 
499. 

25  Bensley,  A.,  Doyle,  R.,  Lyon,  L.  &  Robinson,  S.  Basic  Proficiencies  in  Critical  Thinking,  Frostburg  State 
University,  Frostburg,  MD,  2006,  p.3. 
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Dispositions  that  Explain  Individual  Differences  in  Cognition :  Scholars  who  explored 
individual  differences  in  cognition  tended  to  focus  on  a  single  disposition  at  a  time  rather  than  a 
list  of  thinking  dispositions.  Their  work  generally  described  the  disposition,  researched  the 
impact  of  the  disposition  on  an  individual’s  ability  to  think  effectively,  and  developed 
assessments  for  the  disposition.  I  found  the  following  dispositions  to  be  the  most  frequently 
referenced  in  the  literature: 

•  Need  for  cognition  (NFC)  -  The  disposition  to  engage  in  and  eniov  effortful 

26 

cognitive  activities.” 

•  Need  for  cognitive  closure  -  The  disposition  to  try  for  cognitive  closure  on  a  topic, 
sometimes  resulting  in  premature  closure  before  a  topic  has  been  thoroughly 

97 

investigated. 

•  Actively  open-minded  thinking  -  The  disposition  to  seek  infonnation,  including 

infonnation  that  disconfirms  a  belief;  the  willingness  to  change  a  belief  in  the  face 

28 

of  contradictory  evidence;  an  avoidance  of  bias. 

•  Ambiguity  tolerance/intolerance  -  High  tolerance  for  ambiguity  is  the  disposition  to 
seek  out  ambiguity,  enjoy  ambiguity,  and  excel  in  the  performance  of  ambiguous 
tasks.29 

•  Dogmatism  -  The  disposition  to  hold  a  relatively  closed  cognitive  organization  of 
beliefs  which  provides  a  framework  for  patterns  of  intolerance  toward  others.30 

T 1 

Dogmatism  has  also  been  described  as  close  minded  and  inflexible. 

T9 

•  Rigidity/rigid  thinking  -  The  disposition  to  resist  changing  beliefs  or  habits. 

Making  Sense  of  the  Many  Thinking  Disposition  Examples :  Of  all  the  thinking  disposition 
lists  shown  in  Tables  1  and  2  and  the  individual  dispositions  described  in  the  previous  section, 
which  list  or  individual  disposition  is  the  correct  one  for  any  cognitive  activity?  We  do  not  have 
a  definitive  answer  for  this  question.  Two  factors  should  be  considered: 

1 .  An  organization  should  detennine  which  specific  individual  thinking  dispositions 
are  most  beneficial  to  the  types  of  cognitive  work  needed  by  that  organization. 
Which  items  appear  to  be  most  important  to  the  organization’s  cognitive  work? 
What  terminology  associated  with  the  thinking  disposition  is  understood  by 
members  of  the  organization? 


26  Cacioppo,  J.T.,  Petty,  R.E.,  Feinstein,  J.A.  &  Jarvis,  W.B.G.  "Dispositional  Differences  in  Cognitive  Motivation: 
The  Life  and  Times  of  Individuals  Varying  in  Need  for  Cognition,"  Psychological  Bulletin,  Vol.  119,  No.  2,  1996,  p. 
197. 

27  Kruglanski,  A.W.  &  Webster,  D.M.  "Motivated  Closing  of  the  Mind:  "Seizing"  and  "Freezing","  Psychological 
Review,  Vol.  103,  No.  2, 1996,  p.  264. 

28  Stanovich,  K.E.  &  West,  R.F.  "Reasoning  Independently  of  Prior  Belief  and  Individual  Differences  in  Actively 
Open-Minded  Thinking,"  Journal  of  Educational  Psychology’,  Vol.  89,  No.  2,  1997,  p.  346. 

29  MacDonald,  A.P.  "Revised  Scale  for  Ambiguity  Tolerance:  Reliability  and  Validity,"  Psychological  Reports,  Vol. 
26,  No.  3, 1970,  p.  791. 

30  Rokeach,  M.,  McGovney,  W.C.  &  Denny,  M.R.  "A  Distinction  between  Dogmatic  and  Rigid  Thinking,"  Journal 
of  Abnormal  and  Social  Psychology’,  Vol.  51,  No.  1,  1955,  p.  87. 

31  Shearman,  S.M.  &  Levine,  T.R.  "Dogmatism  Updated:  A  Scale  Revision  and  Validation,"  Communication 
Quarterly,  Vol.  54,  No.  3,  2006,  p.  275. 

32  Rokeach,  M.,  McGovney,  W.C.  &  Denny,  M.R.  "Dogmatic  Thinking  Versus  Rigid  Thinking:  An  Experimental 
Distinction,"  in  The  Open  and  Closed  Mind,  ed.  M.  Rokeach,  Basic  Books,  New  York,  NY,  1960,  p.  183. 
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2.  Can  the  organization  assess  an  individual’s  dispositional  strength  for  any  list  or 
individual  thinking  disposition?  Only  some  of  the  specific  thinking  dispositions 
have  assessments  designed  to  detennine  an  individual’s  strength  of  inclination. 
Section  4.3  contains  information  on  the  different  assessments  available  for  thinking 
dispositions. 

This  list  of  thinking  dispositions,  synthesized  from  the  many  thinking  dispositions  found  during 
my  research,  appears  to  be  of  benefit  to  intelligence  analysis: 

•  Enjoyment  of  cognitive  effort  (NFC),  disposition  to  think  deeply) 

•  Intellectual  curiosity  -  seek  information,  inclination  to  look  for  evidence  (truth¬ 
seeking,  inquisitive) 

•  Open-mindedness  (actively  open-minded  thinking,  fair-mindedness;  not  dogmatic) 

•  Intellectual  honesty  (respect  for  other  viewpoints,  intellectual  integrity) 

•  Tolerance  for  ambiguity  -  inclination  to  wait  for  appropriate  evidence  before 
making  judgments 

•  Focus  on  evidence  -  reach  conclusions  only  after  sufficient  evidence  is  found 

•  Persistent  (intellectual  perseverance) 

•  Systematic  (makes  plans  and  is  strategic,  metacognitive) 

4.1.3.  Related  Concepts  Describing  Affective  or  Motivational  Aspects  of  Cognition 

Psychologists  are  interested  in  the  topic  of  individual  differences  in  personality  and  in  cognition. 
Because  thinking  dispositions  are  one  contributor  to  individual  differences,  they  are  sometimes 
grouped  with  or  connected  to  other  contributors  to  individual  differences. 

Scholars  in  different  fields  use  a  number  of  different  terms  to  describe  the  non-skills  attributes  of 
thinking.  In  my  research  I  found  the  following  terms  used  to  describe  some  aspect  of  individual 
differences  regarding  approaches  to  thinking: 

•  Cognitive  style/thinking  style/intellectual  style 

•  Habits  of  mind 

•  Intellectual  character 

A  large  number  of  scholars  have  considered  cognitive,  thinking,  or  intellectual  styles  to  be 
similar  to  thinking  dispositions.  These  approaches  to  thinking  are  of  interest  not  only  to  scholars 
focused  on  improving  thinking,  but  also  to  scholars  interested  in  teaching  and  learning.  Many 
have  connected  cognitive  styles  with  learning  styles. 

Krouter  described  cognitive  style  as: 

Cognitive  style  refers  to  the  manner  in  which  individuals  acquire  and  process 
information.  Hansen  (1995)  argued  that  “Cognitive  style  measures  do  not  indicate  the 
content  of  the  information  but  simply  how  the  brain  perceives  and  processes  the 
information.”33 


33  Kroutter,  P.  &  Witkin,  H.A.  Field-Dependence  Vs.  Field-Independence  Theory,  2008,  p.  2. 
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I  found  a  variety  of  models  of  thinking/cognitive  styles  during  my  research.  These  models 
include: 

•  Mental  self  government34 

•  Meyers  Briggs  Types35 

•  Herrmann  Brain  Dominance. 

'in 

•  Field  dependence-independence 

After  reviewing  the  literature  I  concluded  that  cognitive  styles  differ  from  thinking  dispositions. 
Cognitive  styles  focus  on  how  someone  approaches  thinking;  thinking  dispositions  focus  on 
whether  someone  willingly  engages  in  thinking.  Although  thinking  styles  may  provide  valuable 
insight  into  effective  analysis,  they  are  a  separate  theoretical  construct  from  thinking 
dispositions  and  are  of  no  further  interest  to  this  study. 

Rather  than  cognitive  styles  or  thinking  dispositions,  some  scholars  wrote  about  thinking  habits, 
specifically  habits  of  the  mind.  Arthur  Costa  clearly  linked  habits  of  the  mind  with  thinking 
dispositions: 

A  "Habit  of  Mind”  means  having  a  disposition  toward  behaving  intelligently  when 
confronted  with  problems,  the  answers  to  which  are  not  immediately  known.  When 
humans  experience  dichotomies,  are  confused  by  dilemmas,  or  come  face  to  face  with 
uncertainties— our  most  effective  actions  require  drawing  forth  certain  patterns  of 
intellectual  behavior.  When  we  draw  upon  these  intellectual  resources,  the  results  that 
are  produced  through  are  more  powerful,  of  higher  quality  and  greater  significance  than  if 
we  fail  to  employ  those  patterns  of  intellectual  behaviors.39 

Costa  identified  16  habits  of  the  mind  that  enable  someone  to  behave  intelligently: 

1 .  Persisting 

2.  Managing  Impulsivity 

3.  Listening  To  Others — With  Understanding  and  Empathy 

4.  Thinking  Flexibly 

5.  Thinking  About  our  Thinking  (Metacognition) 

6.  Striving  For  Accuracy  and  Precision 

7.  Questioning  and  Posing  Problems 

8.  Applying  Past  Knowledge  to  New  Situations 

9.  Thinking  and  Communicating  with  Clarity  and  Precision 

10.  Gathering  Data  through  All  Senses 


34  Sternberg,  R.J.  "Mental  Self-Government:  A  Theory  of  Intellectual  Styles  and  their  Development,"  Human 
Development,  Vol.  31,  No.  4,  1988,  p.  197. 

35  Krizan,  L.  Intelligence  Essentials  for  Everyone,  Joint  Military  Intelligence  College,  Washington,  D.C.,  1999,  p. 
58. 

36  Leonard,  D.  &  Straus,  S.  "Putting  Your  Company's  Whole  Brain  to  Work,"  Harvard  Business  Review’,  Vol.  75, 
No.  4,  1997,  p.  115. 

’7  Kroutter  &  Witkin,  p.  2. 

38  The  Intelligence  Community  has  investigated  the  link  between  intelligence  analysis  and  some  of  these  styles. 

39  Costa,  A.L.  &  Kallick,  B.  Describing  16  Habits  of  Mind,  The  Institute  for  Habits  of  Mind,  [undated],  p.  1. 
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1 1 .  Creating,  Imagining,  and  Innovating 

12.  Responding  with  Wondennent  and  Awe 

13.  Taking  Responsible  Risks. 

14.  Finding  Humor 

15.  Thinking  Interdependetly 

16.  Learning  Continuously40 

This  specific  list  of  habits  of  the  mind  does  overlap  with  the  thinking  dispositions  identified  in 
Tables  1  and  2,  but  also  includes  items  such  as  finding  humor  that  do  not  necessarily  relate  to 
critical,  rational,  and  good  thinking  dispositions. 

James  McTighe  also  described  thinking  habits  that  are  very  similar  to  our  list  of  thinking 
dispositions. 

Effective  problem  solvers,  in  contrast,  view  problems  as  challenges  and  are  persistent  in 
seeking  solutions.  If  a  particular  strategy  is  unsuccessful,  they  take  a  different  approach. 
Other  attitudes  and  habits  of  effective  thinking  are  well  known:  open-mindedness, 
flexibility,  ability  to  defer  immediate  judgment,  attention  to  detail.41 

Ron  Ritchart  described  the  importance  of  intellectual  character  in  his  book  of  the  same  name. 
His  theory  of  intellectual  character  is  founded  upon  thinking  dispositions.  Ritchart  has  worked 
closely  with  Perkins,  Tishart,  and  Jay  who  developed  the  list  of  dispositions  that  lead  to  good 
thinking  found  in  Table  1.  Ritchart  defined  intellectual  character  as: 

As  an  overarching  construct,  the  notion  of  intellectual  character  can  be  understood  only 
in  terms  of  the  thinking  dispositions  that  give  it  shape  and  meaning.42 

While  all  of  the  concepts  related  to  thinking  dispositions  discussed  in  this  section  are  interesting, 
I  have  concluded  that  they  do  not  appear  to  contribute  to  answering  the  questions  posed  by  this 
study.  My  research  confirmed  that  the  term  thinking  dispositions  is  widely  understood  and 
therefore  is  a  sufficient  conceptual  framework  for  describing  an  individual’s  sensitivity  and 
inclination  to  engage  cognitively. 

4.1.4.  How  Have  Thinking  Dispositions  Been  Understood  Historically? 

The  term  thinking  disposition  has  been  in  use  for  a  very  long  time.  Surprisingly,  the  tenn’s 
meaning  has  remained  relatively  consistent.  Google  enables  you  to  find  books  from  specific  time 
periods  that  contain  a  specified  term.  Using  this  feature  I  found  books  from  the  1800s  that 
included  thinking  disposition.  Indeed,  one  book  contained  a  translation  from  Homer’s  Iliad 
which  used  the  term  thinking  disposition  similarly  to  its  usage  in  this  report. 

As  between  lions  and  men  there  are  no  faithful  covenants, — 

Nor  have  wolves  and  lambs  a  same-thinking  disposition, 

But  perpetually  are  plotting  evil  to  each  other;43 


40Costa  &  Kallick,  pp.  2-12 

41  McTighe,  J.J.  "Teaching  for  Thinking,  of  Thinking,  and  about  Thinking,"  in  Thinking  Skills  Instruction:  Concepts 
and  Techniques,  eds.  M.  Heiman  &  J.  Slomianko,  National  Education  Association,  Washington,  D.C.,  1987,  p.  30. 

42  Ritchhart,  R.  Intellectual  Character:  What  It  Is,  Why  It  Matters,  and  How  to  Get  It,  Jossey-Bass,  San  Francisco, 
CA,  2002,  p.  19. 
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Dewey  was  the  most  frequently-referenced  author  when  scholars  highlighted  the  historical  usage 
of  the  tenn  thinking  disposition.  This  quote  from  the  1933  version  of  How  We  Think  was  cited 
frequently: 

Knowledge  of  the  methods  alone  will  not  suffice;  there  must  be  the  desire,  the  will,  to 
employ  them.  This  desire  is  an  affair  of  personal  disposition.44 

In  1941  Edward  Glaser  believed  thinking  dispositions  were  sufficiently  important  that  he 
included  them  as  part  of  his  definition  of  critical  thinking: 

The  ability  to  think  critically,  as  conceived  in  this  volume,  involves  three  things:  (1)  an 
attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that 
come  within  the  range  of  one's  experiences,  (2)  knowledge  of  the  methods  of  logical 
inquiry  and  reasoning,  and  (3)  some  skill  in  applying  those  methods.45 

A  common  understanding  of  the  term  thinking  disposition  continues  to  the  present  time.  Kelly 
Ku  provided  this  explanation  in  2009: 

The  disposition  to  think  critically  includes  the  motivation  of  a  person  (i.e.,  a  matter  of 
choosing  to  engage  in  effortful  thinking  or  not)  and  it  accounts  for  how  critical  thinking 
is  triggered,  “good  timing  -  attempting  the  right  kind  of  thinking  at  the  right  moment” 
(Perkins  &  Ritchhart,  2004,  p.  352).  In  other  words,  what  makes  a  good  thinker  is  now  a 
question  that  “must  be  answered  as  much  in  terms  of  people’s  attitudes,  motivations, 
commitments,  and  habits  of  mind  as  in  terms  of  their  cognitive  abilities”  (Perkins  & 
Ritchhart,  2004,  p.  352).46 

All  of  the  research  I  conducted  appears  to  support  the  impression  that  a  consistent  understanding 
of  thinking  dispositions  has  existed  for  a  remarkably  long  period  of  time.  This  consistency  lends 
credibility  to  the  possibility  of  using  thinking  dispositions  as  part  of  the  selection  and 
development  activities  of  intelligence  analysts. 

4.1.5.  Summary:  What  Are  Thinking  Dispositions? 

Thinking  dispositions  represent  an  individual’s  inclination  toward  thinking  .  An  individual’s 
strength  in  positive  or  productive  thinking  dispositions  identify  whether  someone  is  likely  to 
notice  when  effortful  thinking  is  required  and  the  degree  of  inclination  to  make  the  effort  when 
thinking  is  required.  This  is  a  stable  understanding  of  the  concept  of  thinking  dispositions  over 
time,  across  academic  disciplines,  and  for  different  types  of  thinking. 

Those  thinking  dispositions  that  seem  to  be  applicable  to  intelligence  analysts  might  include: 

•  Enjoyment  of  cognitive  effort 

•  Intellectual  curiosity 

•  Open-mindedness 


4j  Wilson,  J.  Essays  Critical  and  Imaginative  Volume  IV:  Homer  and  His  Translators,  William  Blackwood  and 
Sons,  Edinburgh  and  London,  1857. 

44  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  p.  30. 

45  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  pp. 
5-6. 

4,1  Ku,  K.Y.L.  "Assessing  Students'  Critical  Thinking  Performance:  Urging  for  Measurements  using  Multi-Response 
Format,"  Thinking  Skills  and  Creativity’,  Vol.  4,  No.  1,  2009,  p.  71. 
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•  Intellectual  honesty 

•  Tolerance  of  ambiguity 

•  Focus  on  evidence 

•  Persistent 

•  Systematic 

With  a  clear  understanding  of  what  thinking  dispositions  are  and  the  kinds  of  thinking 
dispositions  that  might  be  useful  for  intelligence  analysis,  we  turn  our  attention  to  why  they  are 
important.  What  impact  do  thinking  dispositions  have  on  cognitive  perfonnance  and  results? 
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4.2  Thinking  Dispositions  and  Cognitive  Performance 

This  section  of  the  report  addresses  the  impact  of  thinking  dispositions.  We’ll  explore: 

•  How  do  thinking  dispositions  affect  cognitive  performance? 

o  How  do  individuals  use  their  thinking  dispositions  while  performing 
cognitive  tasks? 

o  Does  the  use  and  influence  of  thinking  dispositions  vary  depending  on  the 
nature  of  the  cognitive  task? 

A  number  of  scholars  have  conducted  experiments  to  understand  the  impact  that  thinking 
dispositions  have  on  cognitive  performance.  Their  research  has  demonstrated  that  thinking 
dispositions  do  have  a  significant  impact  on  cognitive  performance. 

In  the  previous  section,  we  noted  the  different  approaches  scholars  took  to  defining  thinking 
dispositions.  These  different  approaches  continued  in  the  studies  and  experiments  reported  in 
this  section.  We’ll  explore  the  impact  of  thinking  dispositions  on  cognitive  performance  from 
two  perspectives: 

1 .  Scholars  who  addressed  the  question  of  what  makes  an  effective  thinker  by 
evaluating  the  impact  thinking  dispositions  in  general  have  on  cognitive 

performance. 

2.  Scholars  who  addressed  the  question  of  what  causes  individual  differences  in 
cognitive  effort  and  effectiveness  by  evaluating  the  impact  of  one  or  a  few  specific 
thinking  dispositions. 

As  part  of  the  explanation  of  different  studies  and  experiments,  I  mention  assessments 
researchers  used  to  evaluate  the  strength  of  an  individual’s  thinking  dispositions.  The 
assessments  are  explained  more  fully  in  Section  4.3  of  this  report. 

4.2.1.  General  Experiments  into  the  Impact  of  Thinking  Dispositions  on  Cognitive 
Performance 

Arthur  Whimbey  described  results  of  research  conducted  by  Benjamin  Bloom  and  Lois  Broder  in 
1950.  Although  Bloom  and  Broder  used  the  terms  behavior  and  habit  rather  than  disposition,  it  is 
clear  that  the  report  was  describing  the  difference  between  individuals  with  strong,  positive 
thinking  dispositions  and  individuals  with  weak  or  negative  thinking  dispositions.  Bloom  and 
Broder  were  describing  differences  in  such  thinking  dispositions  as  enjoyment  of  cognitive 
effort,  intellectual  curiosity,  and  being  systematic  and  persistent  in  intellectual  efforts. 

First,  they  asked  students  with  both  high  and  low  scores  on  aptitude  tests  to  think  aloud 
while  solving  academic  reasoning  problems.  They  found  that  the  problem-solving 
behavior  of  low-aptitude  college  students  was  characteristically  different  from  that  of 
high-aptitude  students. 

The  cognitive  profde  of  low-aptitude  college  students  had  two  prominent  features.  First, 
there  was  one-shot  thinking,  rather  than  extended,  sequential  construction  of 
understanding.  Second,  there  was  a  willingness  to  allow  gaps  of  knowledge  to  exist,  an 
attitude  of  indifference  towards  achieving  an  accurate  and  complete  comprehension  of 
situations  and  relations.  These  students  were  mentally  careless  and  superficial  in  solving 
problems.  They  often  rushed  through  instructions  or  even  skipped  them,  and  selected  a 
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wrong  answer  on  the  basis  of  a  feeling  or  a  guess.  They  were  almost  completely  passive 
in  their  thinking,  taking  little  time  to  consider  a  question  or  break  it  down  into  its 
component  parts. 

High-aptitude  students,  by  contrast,  made  an  active  attack  on  problems.  When  a  question 
was  initially  unclear,  they  employed  a  lengthy  sequential  analysis  in  arriving  at  an 
answer.  They  began  with  what  they  understood  of  the  problem,  drew  on  other 
information,  and  carefully  proceeded  through  a  series  of  steps  that  brought  them  to  a 
conclusion. 

Having  isolated  the  habitual  thinking  deficiencies  of  low-aptitude  students,  Bloom  and 
Broder  undertook  to  develop  a  remedial  training  program.47 

Philosopher  Stephen  Norris  conducted  a  series  of  experiments  to  determine  whether  performance 
on  critical  thinking  skills  tests  were  an  accurate  reflection  of  critical  thinking  abilities.  He 
theorized  that  performance  on  these  tests  might  be  constrained  by  weak  thinking  dispositions. 

To  test  this  theory  he  created  modified  versions  of  the  tests  to  simulate  stronger  thinking 
dispositions. 

The  modifications  were  not  designed  to  teach  critical  thinking,  that  is,  to  provide 
examinees  critical  thinking  abilities  they  did  not  have.  The  guidelines  and  hints 
suggested  to  examinees  what  they  might  do,  but  did  not  instruct  them  in  how  to  do  it. 
Thus,  if  they  did  not  have  the  appropriate  abilities,  the  guidelines  and  hints  would  be  of 
no  value.  However,  if  they  did  have  the  abilities,  but  not  the  dispositions  to  use  them, 
then  the  guidelines  and  hints  could  serve  in  the  place  of  dispositions,  as  surrogate 
dispositions,  and  if  effective  would  lead  to  improved  performance.48 

He  found  mixed  results.  Participants  showed  no  difference  in  outcome  on  some  of  the  multiple 
choice  or  short  answer  skills  tests  when  disposition-oriented  hints  were  provided.  However,  on 
some  of  the  tests,  the  dispositional  hints  did  cause  significant  improvement. 

Second,  performance  levels  on  some  critical  thinking  tests  underestimate  examinees' 
critical  thinking  abilities  in  an  area  because  examinees'  disposition  to  think  critically  falls 
short  of  their  ability  to  do  so.  Both  the  Ennis-Weir  and  the  multiple-choice  version  of  the 
Observations  test  showed  positive  gains  for  the  hints  version  over  the  original  and 
guidelines  versions.49 

Educators  Perkins  and  Ritchart  also  explored  the  impact  of  thinking  dispositions  in  general  on 
cognitive  effort.  For  example,  in  one  experiment  they  found: 

In  later  research,  the  interviewer  pushed  subjects  to  elaborate  their  arguments  on  both 
sides  further.  When  it  appeared  that  a  subject  had  no  more  to  say,  the  interviewer  then 
asked  the  subject  point  blank  to  identify  weaknesses  in  his  or  her  argument  and  to 
elaborate  the  other  side  of  the  case.  Subjects  could  do  so  readily.  Most  dramatically, 
when  directly  prompted,  subjects  increased  points  mentioned  on  the  other  side  of  the  case 


47  "Conversation  with  Arthur  Whimbey,"  in  Thinking  Skills  Instruction:  Concepts  and  Techniques,  eds.  M.  Heiman 
&  J.  Slomianko,  National  Education  Association,  Washington,  D.C.,  1987,  pp.  161-162. 

'  '  Norris,  S.P.  "The  Meaning  of  Critical  Thinking  Test  Performance:  The  Effects  of  Abilities  and  Dispositions  on 
Scores,"  in  Critical  Thinking:  Current  Research,  Theory  and  Practice,  ed.  D.J.  Fasko,  Hampton  Press,  Cresskill,  NJ, 
2003,  p.  321. 

49  Norris,  p.  325. 
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by  an  average  of  700%  (Perkins  et  al.,  1991).  The  data  showed  that  subjects  generally 
did  not,  but  easily  could,  examine  the  other  side  of  the  case  with  care.  It  implicated  an 
important  role  for  dispositions  in  thinking:  People  in  trend  were  capable  of,  but  not 
generally  disposed  to,  critique  their  own  arguments  or  examine  the  other  side  of  the 

50 

case. 

The  authors  also  explored  the  specific  contribution  made  by  each  component  in  their  three-part 
understanding  of  thinking  dispositions:  sensitivity,  inclination,  and  ability 

The  most  important  finding,  confirmed  over  and  over  again  in  our  work,  showed  that 
dispositional  considerations  more  than  abilities  limited  thinking.  Sensitivity  was  by  far 
the  greatest  bottleneck,  followed  by  inclinations.51 

Psychologists  Dweck  and  Leggett  conducted  a  series  of  experiments  to  understand  the  impact  of 
disposition  and  situation  on  cognitive  performance.  In  the  experiments  summarized  in  their  1988 
article  entitled  “A  Social-Cognitive  Approach  to  Motivation  and  Personality”  they  specifically 
investigated  the  behavior  of  students  who  have  different  goals  in  a  learning  situation.  After 
assessing  the  students’  performance,  the  investigators  then  provided  guidance  to  override  the 
students’  goals  in  order  to  create  situational  changes.  They  found: 

First,  our  research  has  clearly  shown  that  both  situational  variables  and  dispositional 
variables  play  important  roles  in  producing  behavior.  We  have  experimentally  induced 
goals  and  behavior  patterns  by  manipulating  situational  variables  (Dweck,  Davidson, 
Nelson,  &  Enna,  1978;  Elliott  &  Dweck,  1988),  but  we  have  also  predicted  goal  choice 
and  behavior  patterns  by  measuring  existing  dispositional  variables  (e.g.,  implicit 
theories:  Bandura  &  Dweck,  1985;Leggett,  1985).  A  view  that  integrates  these  findings 
is  one  in  which  dispositions  are  seen  as  individual  difference  variables  that  determine  the 
a  priori  probability  of  adopting  a  particular  goal  and  displaying  a  particular  behavior 

52 

pattern,  and  situational  factors  are  seen  as  potentially  altering  these  probabilities. 

Several  different  scholars  attempted  to  find  correlations  between  results  on  the  CCTDI  and 
academic  performance  but  were  unable  to  find  a  direct  correlation.  For  example,  shortly  after  he 
completed  development  of  the  CCTDI,  Peter  Facione  investigated  its  usefulness  in  several 
different  studies. 

The  purpose  of  the  current  research  is  to  explore  conceptually  the  disposition  toward 
critical  thinking,  and  to  report  preliminary  empirical  explorations  concerning  whether 
college  students,  display  that  disposition.53 

Facione’s  findings  include: 

The  sample  described  in  Table  1  includes  only  entering  freshmen  who  had  not  yet 
experienced  college  level  instruction.  Only  13%  of  these  567  new  college  students  were 


50  Perkins,  D.  &  Ritchhart,  R.  "When  is  Good  Thinking?"  in  Motivation,  Emotion,  and  Cognition:  Integrative 
Perspectives  on  Intellectual  Functioning  and  Development,  eds.  D.Y.  Dai  &  R.J.  Sternberg,  L.  Erlbaum  Associates, 
Mahwah,  NJ,  2004,  pp.  361-362. 

51  Perkins  &  Ritchhart,  p.  363. 

5"  Dweck,  C.S.  &  Leggett,  E.L.  "Social-Cognitive  Approach  to  Motivation  and  Personality,"  Psychological  Review, 
Vol.  95,  No.  2,  1988,  p.269. 

53  Facione,  P.A.,  Sanchez,  C.  A.  &  Facione,  N.C.  Are  College  Students  Disposed  to  Think?,  California  Academic 
Press,  Millbrae,  CA,  1994,  p.  3. 
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positive  on  all  seven  CT  dispositional  scales  of  the  CCTDI.  The  other  87%  were 
disposed  against  at  least  one  of  the  seven  aspects  of  the  overall  disposition  toward  CT.5 

A  comparison  of  graduate  students  and  undergraduates  at  an  institution  did  show  stronger 
thinking  dispositions  for  graduates,  but  the  study  could  not  identify  the  cause: 

The  profile  of  the  graduate  students  is  stronger  in  each  dimension,  which  may  reflect 
genuine  growth  or  which  may  be  an  artifact  of  selection  to  graduate  school.55 

In  a  study  of  nursing  students  from  over  50  different  nursing  programs,  Peter  and  Noreen 
Facione  attempted  to  understand  the  relationship  between  critical  thinking  and  nursing. 

Research  questions  included: 

This  was  a  real  world  investigation  of  a  process  in  action,  not  a  planned  quasi¬ 
experiment.  The  principal  investigator  and  on-site  investigators  often  talked  at  length 
about  the  kinds  of  data  that  would  be  needed  to  answer  complex  questions  like  "What  is 
the  relationship  between  critical  thinking  skills  and  dispositions  and  the  development  of 
expertise  in  nursing  practice?"  or  "What  kinds  of  data  would  be  most  informative  of  a 
particular  learning  outcome?"56 

Unfortunately,  the  data  from  this  study  varied  from  institution  to  institution,  making  it  difficult  to 
draw  any  useful  conclusions.  The  researchers  were  not  able  to  collect  data  upon  entry  as  well  as 
upon  exit  from  the  programs  to  compare  the  results  nor  were  they  able  to  make  connections  to 
academic  performance. 

The  analyses  of  correlations  between  CT  dispositions  and  academic  achievement 
measures  were  surprisingly  mixed.  As  expected  there  were  no  significant  relationships 
between  CCTDI  scales  and  Scholastic  Aptitude  Test  (SAT)  math  scores,  not  even  in  the 
case  of  systematicity,  where  we  might  have  expected  some  modest  degree  of  relationship. 
But  we  did  see  a  relationship  between  several  of  the  dispositional  scores  and  both  the 
SAT  verbal  and  the  American  College  Test  (ACT).  The  sample  for  which  these  values 
were  available  is  quite  small.  We  prefer  to  simply  report  these  results  and  to  defer 
interpretation  of  them  at  this  time.  If  they  are  replicated  in  other  samples,  it  would  then 
be  important  to  try  to  determine  what  is  being  measured  by  the  SAT  verbal  and  the  ACT 
that  could  be  related  to  the  CCTDI  scales. 

Our  desire  to  examine  CT  dispositions  in  relation  to  Grade  Point  Average  (GPA) 
variables  was  all  but  impossible.57 

Finally,  the  investigators  reported  an  inability  to  answer  the  question  of  connection  between 
thinking  dispositions  and  nursing  expertise. 

We  were  similarly  unable  to  examine  the  relationship  between  scores  on  the  CCTDI 
scales  and  a  successful  passage  on  the  National  Council  Licensure  Examination 


54  Facione  et  al.  p.  10. 

53  Facione  et  al.  p.  10. 

56  Facione,  N.C.  &  Facione,  P.A.  Critical  Thinking  Assessment  in  Nursing  Education  Programs:  An  Aggregate  Data 
Analysis,  California  Academic  Press,  Millbrae,  CA,  1997,  pp.  5-6. 

57  Facione  &  Facione,  p.  73. 


24 


Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-20 14-2455;  Cleared  21  May  2014 


(NCLEX)  examination  due  to  too  few  individuals  for  whom  NCLEX  scores  were 

SR 

contributed  to  the  aggregate  data  set. 

Walsh  and  Seldomridge  experienced  similar  challenges  in  a  2006  review  of  critical  thinking  in 
nursing  education.  As  a  result  of  their  research  they  recognized  the  need  to  have  a  better 
understanding  of  what  was  to  be  measured  and,  perhaps,  a  different  approach  to  measuring  it. 

Although  demonstrating  modest  gains  in  critical  thinking  dispositions  by  some  cohorts, 
data  collected  from  1997  to  2002  showed  no  consistency  of  pattern.  While  some  cohorts 
saw  gains,  subsequent  cohorts  saw  losses,  and  several  years  had  essentially  no  change. 
Throughout  this  period,  the  usual  academic  indicators  (e.g.,  SAT/ACT  data,  cumulative 
grade  point  average,  science  grades)  remained  the  same.  The  data  on  critical  thinking 
skills  were  likewise  inconclusive  and  difficult  to  explain.  This  raised  concerns  about 
both  the  definition  of  critical  thinking  and  the  utility  of  using  standardized  instruments  to 
measure  critical  thinking.59 

4.2.2.  Experiments  That  Study  the  Impact  of  Specific  Thinking  Dispositions 

The  researchers  discussed  in  this  section  of  the  report  are  psychologists  interested  in 
understanding  the  causes  of  or  contributors  to  individual  differences  in  cognitive  processing. 

The  experiments  they  conducted  tended  to  be  much  more  focused  with  clearer  investigational 
goals.  As  a  consequence,  the  results  appeared  to  be  more  useful. 

IQ  is  generally  accepted  as  one  primary  cause  of  individual  cognitive  differences.  Individuals 
with  higher  IQs  have  the  capacity  to  think  better  or  more  effectively  than  individuals  with  lower 
IQs.  However,  researchers  have  found  that  this  does  not  explain  all  individual  differences  in 
thinking  ability.  Stanovich  described  the  challenge  this  way: 

Intelligence  tests  are  thus  radically  incomplete  as  measures  of  cognitive  functioning. 
Because  of  their  vast  influence,  IQ  tests  have  both  explicitly  and  implicitly  defined,  for 
the  layperson  and  psychologist  alike,  what  cognitive  attributes  to  value.  These  are 
important  abilities,  to  be  sure,  but  the  tests  leave  out  huge  domains  of  cognitive 
functioning.60 

As  a  reminder,  I  found  the  following  dispositions  to  be  the  most  frequently  referenced  in  the 
literature.  I’ll  discuss  the  impact  of  each  of  these  thinking  dispositions  individually. 

•  Dogmatism 

•  Rigidity/rigid  thinking 

•  Need  for  cognition 

•  Need  for  cognitive  closure 

•  Actively  open-minded  thinking 

•  Ambiguity  tolerance/intolerance 


58  Facione  &  Facione,  p.  73. 

59  Walsh,  C.M.  &  Seldomridge,  L.A.  "Critical  Thinking:  Back  to  Square  Two,"  Journal  of  Nursing  Education,  Vol. 
45,  No.  6,  2006,  pp.  212-213. 

60  Stanovich,  K.E.  What  Intelligence  Tests  Miss:  The  Psychology  of  Rational  Thought,  Yale  University  Press,  New 
Flaven,  CT,  2009,  p.  xi. 
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Dogmatism  and  Rigid  Thinking:  The  earliest  experimental  results  that  I  found  came  from  a 
1955  experiment  conducted  by  psychologists  Rokeach  et  al.  concerning  dogmatism  and  rigid 
thinking.  Rokeach  defined  dogmatism  as  a  closed  system  of  beliefs  and  rigidity  as  the  way  a 
person  addresses  a  specific  task  or  problem.  The  experiment  consisted  of  providing  a 
challenging  problem  to  the  research  subjects  that  required  them  to  overcome  both  individual 
beliefs  and  systems  of  beliefs.  Subjects  completed  the  thinking  disposition  tools  known  as  the 
Dogmatism  Scale  and  the  Rigidity  Scale. 

Rokeach  described  the  experiment  this  way: 

We  will  attempt  to  demonstrate  experimentally  this  aspect  of  dogmatism  -  and  how  it  is 
distinguished  from  rigidity  -  by  presenting  subjects  (Ss)  with  a  cognitive  task  which,  if  it 
is  to  be  solved,  involves  the  successful  completion  of  two  discrete  stages  of  the 
problemsolving  process:  (a)  the  overcoming  of  several  sets  (or  beliefs,  or  expectancies) 
and  (b)  the  integration  of  new  sets  -  after  the  older  ones  have  been  overcome  -  into  the 
problem  solution.61 

The  findings  demonstrated  that  individuals  with  different  scores  on  the  Rigidity  Scale  and 
Dogmatism  Scale  did  perfonn  differently  in  the  task.  Those  with  a  higher  degree  of  dogmatism 
had  greater  difficulty  integrating  multiple  new  concepts  into  the  solution.  Those  with  a  higher 
degree  of  rigidity  had  greater  difficulty  integrating  single  new  concepts. 

1 .  The  construct  of  dogmatism,  and  its  operational  definition  by  means  of  the  scale 
designed  to  measure  it,  can  now  with  some  empirical  validity  be  said  to  deal  with 
cognitive  systems  of  expectancies,  and  the  extent  to  which  such  systems  are 
organized  in  an  open  or  closed  manner.  This  is  evidenced  by  the  differential 
responses  of  the  High  and  Low  Dogmatic  Groups  to  a  cognitive  task  especially 
designed  to  bring  out  this  distinguishing  property  of  the  construct  of  dogmatism. 
While  high  and  low  scorers  in  dogmatism,  as  has  been  seen,  are  not  found  to  differ 
significantly  from  each  other  in  ease  of  overcoming  single  sets,  they  do  appear  to 
differ  from  each  other  in  ease  of  integrating  several  sets  into  a  new  cognitive 
system.  The  greater  difficulty  manifested  by  the  High  Dogmatic  Group  in 
integrating  newly  arrived  at  sets  into  a  new  system  can  be  seen  as  a  function  of  the 
stronger  operation  of  systems  of  older  sets  organized  into  relatively  more  closed 
systems. 

2.  The  findings  lend  further  empirical  support  to  the  theoretical  distinction  drawn 
between  the  constructs  of  dogmatism  and  rigidity.  As  has  been  shown,  high  and 
low  scorers  on  rigidity  are  differentiated  on  measures  involving  the  overcoming  of 
single  sets  but  not  on  measures  involving  the  integration  of  such  sets. 

3.  Finally,  the  findings  suggest  that  individual  differences  in  personality  organization 
play  an  important  role  in  problem-solving  behavior.  This  is  indicated  by  the  fact 
that  differential  responses  to  the  Denny  Doodlebug  Problem  can  be  systematically 
related  to  differential  responses  to  the  two  personality  scales  employed  in  this 
study.62 


61  Rokeach,  M.,  McGovney,  W.C.  &  Denny,  M.R.  "A  Distinction  between  Dogmatic  and  Rigid  Thinking,"  Journal 
of  Abnormal  and  Social  Psychology,  Vol.  51,  No.  1,  1955,  p.  87. 

62  Rokeach  et  al.  p.92. 
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A  2006  report  on  dogmatism  included  descriptions  of  the  impact  of  dogmatism  in  every  day 
reasoning.  For  example: 

More  recently,  Palmer  and  Klain  (1985)  reported  that  the  individuals  high  in  dogmatism 
attempt  to  avoid  information  that  is  inconsistent  with  their  belief  systems,  and  they  react 
to  inconsistent  information  by  minimizing  or  ignoring  it.  Leone  (1989)  reported  that  as 
the  opportunity  for  thoughts  increases,  dogmatic  people  are  more  likely  to  polarize  their 
attitudes,  and  hold  more  attitude-consistent  beliefs  and  fewer  attitude-inconsistent  beliefs. 
Davies  (1993)  investigated  the  effect  of  dogmatism  on  the  persistence  of  beliefs  when 
facing  discredited  evidence.  Davies  (1998)  also  investigated  the  effect,  of  dogmatism  on 
belief  formation  and  reported  that  high  dogmatics  produced  more  consistent  reason 

/TO 

generation  than  inconsistent  reasons  compared  to  low  dogmatics. 

NFC:  Psychologist  John  Cacioppo  et  al.  have  investigated  the  impact  of  the  thinking  disposition 
known  as  the  NFC  since  the  1980s.  In  a  report  summarizing  the  results  of  multiple  studies  using 
their  NFC  Scale  (a  thinking  disposition  assessment  tool)  they  reported  a  number  of  interesting 
impacts: 

Another  key  set  of  distinctions  between  individuals  high  and  low  in  N  is  that  the  fonner 
enjoy  effortful  reasoning  and  problem  solving  more  and  are  less  stressed  by  cognitively 
effortful  problems,  complex  life  circumstances,  or  cognitively  demanding  tasks.64 

A  meta-analytic  comparison  of  the  overall  difference  in  recall  between  participants  high 
and  low  in  need  for  cognition  across  all  of  these  studies  demonstrated  that,  in  general, 
individuals  high  in  need  for  cognition  recall  more  of  the  information  to  which  they  are 
exposed  than  individuals  low  in  need  for  cognition  (d  =  355,  p  <  .001;  see  Table  3). 65 

In  sum,  research  manipulating  the  quality  of  the  arguments  in  a  persuasive  message 
strongly  indicates  that  individuals  high  in  need  for  cognition  are  more  likely  to  exert  the 
cognitive  effort  necessary  to  process  the  substantive  merits  of  the  information  to  which 
they  are  exposed  than  are  individuals  low  in  need  for  cognition.66 

Several  tests  have  been  reported  of  the  hypothesis  that  individuals  high  in  need  for 
cognition  are  more  likely  to  generate  issue-  or  task-relevant  thoughts  than  individuals  low 
in  need  for  cognition.67 

A  recent  experiment  by  Savia  Coutinho  investigated  the  impact  that  NFC  has  on  intellectual 
performance.  This  study  also  compared  NFC  with  metacognition.  Coutinho  described 
metacognition  this  way: 

Metacognition  refers  to  one’s  ability  to  know  and  regulate  cognitive  processes  (Schraw  & 
Moshman,  1995).  Flavell  (1979)  conceptualized  metacognition  as  “knowledge  and 


Shearman,  S.M.  &  Levine,  T.R.  "Dogmatism  Updated:  A  Scale  Revision  and  Validation,"  Communication 
Quarterly,  Vol.  54,  No.  3,  2006,  p.  276. 

64  Cacioppo,  J.T.,  Petty,  R.E.,  Feinstein,  J.A.  &  Jarvis,  W.B.G.  "Dispositional  Differences  in  Cognitive  Motivation: 
The  Life  and  Times  of  Individuals  Varying  in  Need  for  Cognition,"  Psychological  Bulletin,  Vol.  119,  No.  2,  1996,  p. 
215. 

65  Cacioppo  et  al.  p.  229. 

66  Cacioppo  et  al.  pp.  229-230. 

67  Cacioppo  et  al.  p.  231. 
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cognition  about  cognitive  phenomena."  Simply  stated,  metacognition  is  thinking  about 
your  thinking.68 

The  results  of  the  experiment  confirmed  the  value  of  NFC: 

This  study  examined  the  relationship  between  the  need  for  cognition  and  metacognition 
and  how  these  variables  relate  to  intellectual  task  performance.  Students  completed  items 
from  the  GRE  analytical  subtest  and  this  was  used  as  a  measure  of  intellectual  task 
performance.  Only  the  need  for  cognition  was  a  significant  predictor  of  performance. 
Students  who  had  a  strong  desire  to  understand  and  solve  complex  problems  tended  to 
respond  accurately  to  these  problems.  Metacognition  did  not  appear  to  benefit  students  in 
performance  terms  as  students  with  good  metacognition  did  not  score  significantly  better 
than  students  with  poor  metacognition.69 

Petty  et  al.  summarized  several  decades  of  research  into  the  NFC  in  2009  and  described  the 
impact  of  the  thinking  disposition  this  way: 

At  the  most  basic  level,  NC  affects  the  amount  of  thought  that  goes  into  a  decision.  Thus 
those  high  in  NC  tend  to  think  more  about  available  options  prior  to  making  a  decision 
(Levin,  Huneke,  &  Jasper,  2000)  and  are  more  likely  to  search  for  additional  information 
before  coming  to  a  judgmental  conclusion  (Yang  &  Lee,  1998). 70 

Petty  et  al.  also  noted  that  individuals  high  in  NFC  are  not  immune  to  bias  in  their  thinking. 

Across  a  variety  of  studies,  those  low  in  NC  tend  to  show  greater  amounts  of  bias  when 
this  bias  is  created  by  a  reliance  on  mental  shortcuts.  Alternatively,  when  the  bias  is 
created  through  effortful  thought,  individuals  high  in  NC  tend  to  be  more  strongly 
affected.  When  a  bias  can  come  about  through  either  route,  individuals  both  low  and 
high  in  NC  can  show  the  effect,  but  it  will  be  produced  by  different  mechanisms.71 

NFC  Closure :  The  need  for  cognitive  closure  appears  to  be  a  situation-dependent  thinking 
disposition.  Psychologists  Webster  and  Kruglanski  reported  on  a  number  of  factors  that  have  an 
impact  on  the  NFC  closure: 

Thus,  Kruglanski  and  Freund  (1983)  found  that  elevating  the  need  for  closure  through 
time  pressure  increased  subjects'  tendency  to  succumb  to  primacy  effects  in  impression 
formation,  render  stereotypically  driven  judgments,  and  anchor  judgments  on  initial 
estimates,  all  presumed  to  represent  various  effects  of  the  need  for  closure  on  the 
judgmental  process.  Similar  time  pressure  effects  were  obtained  in  research  by  Freund, 
Kruglanski,  and  Schpitzajzen  (1985),  Heaton  and  Kruglanski  (1991),  Jamieson  and 
Zanna  (1989),  and  Sanbomatsu  and  Fazio  (1990). 

Webster  (1993)  manipulated  the  need  for  closure  through  varying  the  perceived 
attractiveness  of  an  attitude-attribution  task  (Jones  &  Harris,  1967).  When  the  task  was 
perceived  as  unattractive  (rendering  extensive  processing  of  relevant  information  costly), 


68  Coutinho,  S.A.  "The  Relationship  between  the  Need  for  Cognition,  Metacognition  and  Intellectual  Task 
Performance,"  Educational  Research  and  Reviews,  Vol.  1,  No.  5,  2006,  p.  162. 

69  Coutinho  p.  164. 

70  Petty,  R.E.,  Brinol,  P.,  Loersch,  C.  &  McCaslin,  M.J.  "The  Need  for  Cognition,"  in  Handbook  of  Individual 
Differences  in  Social  Behavior,  eds.  M.R.  Leary  &  R.H.  Hoyle,  Guilford  Press,  New  York,  NY,  2009,  p.  322. 

71  Petty  et  al.  p.  323. 
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subjects  were  more  likely  to  exhibit  the  "correspondence  bias"  than  when  the  task  was 
perceived  as  moderately  attractive.  Furthermore,  when  the  task  was  perceived  as  highly 
attractive  (reducing  the  perceived  costs  of  information  processing),  the  tendency  to 

79 

exhibit  the  correspondence  bias  was  all  but  eliminated. 

Webster  and  Kruglanski  summarized  the  findings  from  multiple  experiments: 

In  the  realm  of  social  cognition,  for  example,  the  need  for  closure  was  found  to  affect, 
among  others,  persons'  tendency  to  (a)  use  stereotypes  or  render  judgments  in  a  category- 
based  versus  attribute-based  fashion  (Fiske  &  Neuberg,  1990;  Jamieson  &  Zanna,  1989; 
Kruglanski  &  Freund,  1983,  Study  2),  (b)  exhibit  construct-accessibility  effects  in 
interpreting  ambiguous  information  (E.  P.  Thompson,  Roman,  Moskowitz,  Chaiken,  & 
Bargh,  1993),  (c)  exhibit  correspondence  bias  in  person  perception  (Webster,  1993),  and 
(d)  conduct  social  comparisons  with  similar  versus  dissimilar  others  (Kruglanski  & 
Mayseless,  1987).  In  the  realm  of  social  interactions,  the  need  for  closure  was  found  to 
affect  (a)  persons’  readiness  to  be  persuaded  by  their  partners  (Kruglanski  et  al.,  1993) 
and  (b)  group  members'  tendency  to  react  with  rejection  to  opinion  deviates  (Kruglanski 
&  Webster,  1991).73 

Actively  Open-Minded  (AOT)  Thinking:  Psychologist  Keith  Stanovich  and  colleagues  have 
conducted  experiments  into  the  impact  of  thinking  dispositions  on  cognitive  performance  for 
many  years.  The  experiments  have  repeatedly  demonstrated  that  when  individuals  with  similar 
intelligence  levels  (IQ)  are  faced  with  the  same  problem,  those  who  have  thinking  dispositions 
that  lead  to  stronger  cognitive  effort  perform  better.  An  early  experiment  conducted  by 
Stanovich  and  West  in  1997  focused  on  the  impact  of  open-mindedness.  They  assessed 
individual  biases  on  different  arguments  as  a  separate  step  in  the  experiment.  The  researchers 
then  asked  participants  to  provide  SAT  scores  and  complete  a  vocabulary  test  to  identify  relative 
IQ.  Finally,  participants  were  asked  to  complete  an  assessment  developed  by  the  researchers 
called  the  Thinking  Dispositions  Questionnaire  to  assess  various  dispositional  characteristics. 
Stanovich  and  West  combined  several  of  the  individual  dispositional  characteristics  into  one 
composite  outcome  score  they  called  the  AOT  score: 

Because  several  of  the  scales  displayed  moderate  intercorrelations  (see  Appendix  B),  a 
composite  actively  open-minded  thinking  (AOT)  score  was  formed  by  summing  the 
scores  on  the  Flexible  Thinking,  Openness-Ideas,  and  Openness- Values  scales  and 
subtracting  the  sum  of  the  Absolutism,  Dogmatism,  and  Categorical  Thinking  scales. 
Thus,  high  scores  on  the  AOT  composite  indicate  openness  to  belief  change  and 
cognitive  flexibility,  whereas  low  scores  indicate  cognitive  rigidity  and  resistance  to 
belief  change.74 

Their  results  demonstrated  that  thinking  dispositions  accounted  for  differences  in  cognitive 
performance  independent  of  the  SAT,  their  measure  of  intelligence  in  this  experiment75: 


72  Webster,  D.M.  &  Kruglanski,  A.W.  "Individual  Differences  in  Need  for  Cognitive  Closure,"  Journal  of 
Personality  and  Social  Psychology,  Vol.  67,  No.  6,  1994,  pp.  1049-1050. 

73  Webster  &  Kruglanski,  p.  1061. 

74  Stanovich,  K.E.  &  West,  R.F.  "Reasoning  Independently  of  Prior  Belief  and  Individual  Differences  in  Actively 
Open-Minded  Thinking,"  Journal  of  Educational  Psychology,  Vol.  89,  No.  2,  1997,  p.  347. 

75  Stanovich  and  West  used  a  tool  called  the  argument  evaluation  test,  referred  to  as  the  AET  in  this  quote,  to 
measure  the  research  subjects’  critical  thinking  abilities. 
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The  criterion  variable  in  the  series  of  hierarchical  regression  analyses  presented  in  Table 
2  was  the  beta  weight  for  argument  quality  on  the  AET.  Entered  first  into  the  regression 
equation  was  the  SAT  total  score  which,  not  surprisingly,  given  the  results  displayed  in 
Table  1,  accounted  for  a  significant  proportion  of  variance.4  However,  listed  next  are 
alternative  second  steps  in  the  hierarchical  analysis.  As  the  table  indicates,  the  AOT 
composite  score  accounted  for  significant  variance  in  performance  on  the  AET  even  after 
SAT  scores  were  entered  into  the  equation — as  did  scores  on  the  superstitious  thinking, 
outcome  bias  (a  measure  of  decontextualization),  and  counterfactual  thinking  (a  measure 
of  perspective  switching)  measures.  Thus,  the  linkage  between  thinking  dispositions  and 
performance  on  the  AET  illustrated  in  Table  1  is  not  entirely  due  to  covariance  with 
cognitive  ability.  Various  measures  of  thinking  dispositions  are  predictors,  independent 
of  SAT  scores.76 

Stanovich  and  his  colleagues  have  continued  experiments  into  the  impact  of  thinking  dispositions 
on  cognitive  performance  to  the  present  time.  He  summarized  years  of  experiments  in  his  2010 
book  entitled  What  Intelligence  Tests  Miss:  the  Psychology  of  Rational  Thought. 

In  my  own  laboratory,  we  have  developed  an  argument  evaluation  task  in  which  we 
derive  an  index  of  the  degree  to  which  argument  evaluation  is  associated  with  argument 
quality  independent  of  prior  belief.16  Intelligence  did  in  fact  correlate  with  the  ability  to 
avoid  belief  bias  in  our  task.  Nonetheless,  we  have  consistently  found  that,  even  after 
statistically  controlling  for  intelligence,  individual  differences  on  our  index  of  argument- 
driven  processing  can  be  predicted  by  a  variety  of  thinking  dispositions,  including: 
measures  of  dogmatism  and  absolutism;  categorical  thinking;  flexible  thinking;  belief 
identification;  counterfactual  thinking;  superstitious  thinking;  and  actively  open-minded 
thinking.77 

Ambiguity  Tolerance  (AT)/Intolerance :  In  1995  Furnham  and  Ribchester  summarized  five 
decades  of  research  into  ambiguity  tolerance  in  individuals  and  organizations.  They  noted: 

Ambiguity  tolerance  (AT)  refers  to  the  way  an  individual  (or  group)  perceives  and 
processes  information  about  ambiguous  situations  or  stimuli  when  confronted  by  an  array 
of  unfamiliar,  complex,  or  incongruent  clues.  AT  is  a  variable  that  is  often  conceived  on 
an  unidimensional  scale.  The  person  with  low  tolerance  of  ambiguity  experiences  stress, 
reacts  prematurely,  and  avoids  ambiguous  stimuli.  At  the  other  extreme  of  the  scale, 
however,  a  person  with  high  tolerance  for  ambiguity  perceives  ambiguous 
situations/stimuli  as  desirable,  challenging,  and  interesting  and  neither  denies  nor  distorts 
their  complexity  of  incongruity. 

The  authors  noted  that  some  evidence  exists  that  using  ambiguity  tolerance  during  personnel 
selection  is  valuable: 

The  application  of  an  AT  concept  or  use  of  any  of  the  measures  in  management  selection, 
training,  or  education  is  surprisingly  limited.  Where  good  research  has  been  done  with 
representative  samples  and  robust  and  realistic  dependent  measures,  AT  has  proven  to  be 


76  Stanovich  &  West,  p.  349. 

77  Stanovich,  K.E.  What  Intelligence  Tests  Miss:  The  Psychology  of  Rational  Thought,  Yale  University  Press,  New 
Haven,  CT,  2009,  p.  34. 

78  Furnham,  A.  &  Ribchester,  T.  "Tolerance  of  Ambiguity:  A  Review  of  the  Concept,  its  Measurement  and 
Applications,"  Current  Psychology’,  Vol.  14,  No.  3,  1995,  p.  179. 
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a  highly  predictive  individual  difference  variable.  Clearly,  individuals  with  low  AT 
appear  to  be  more  sensitive  to  stress,  more  risk  averse,  and  more  sensitive  to  particular 
kinds  of  feedback.79 

4.2.3.  Summary:  Thinking  Dispositions  and  Cognitive  Performance 

It  is  clear  that  thinking  dispositions  are  a  trigger  for  cognitive  activity.  When  an  individual  is 
disposed  to  think,  he  or  she  makes  the  effort  to  engage  cognitively.  The  specific  thinking 
dispositions  possessed  by  the  individual  and  the  relative  strength  of  these  individual  dispositions 
influence  the  degree  to  which  an  individual  makes  an  effort  to: 

•  Engage  cognitively  at  all 

•  Override  personal  beliefs  and  biases  to  ensure  an  open-minded  approach  to  the  task 

•  Thoroughly  investigate  a  situation  in  order  to  find  relevant  information  and  to  look 
at  all  sides  of  an  issue 

•  Avoid  premature  closure  of  the  thinking  task  and  to  tolerate  the  lack  of  a  clear 
answer  (ambiguity)  while  investigation  continues 

•  Reach  a  conclusion  only  after  finding  sufficient  evidence  to  support  the  conclusion 

If  we  can  determine  how  to  assess  an  analyst’s  thinking  dispositions,  evaluating  thinking 
dispositions  could  be  a  valuable  part  of  selecting  intelligence  analysts.  In  the  next  section  we’ll 
explore  the  assessment  tools  that  might  be  used  during  intelligence  analysis  selection. 

4.3  Assessing  Thinking  Dispositions 

This  section  of  the  report  addresses  the  question  of  how  to  determine  the  relative  strength  of  an 
individual’s  thinking  dispositions.  We’ll  explore  the  following  questions: 

•  Can  thinking  dispositions  be  assessed? 

•  Do  reliable  and  valid  assessment  methods  exist? 

•  What  evidence  exists  that  supports  the  value  of  thinking  disposition  assessments? 

I  found  that  the  following  assessments  do  make  some  reference  to  thinking  dispositions: 

•  CCTDI 

•  AOT  Scale 

•  Thinking  Dispositions  Questionnaire 

•  NFC  Scale 

•  NFC  Closure  Scale 

•  Dogmatism  Scale 

•  Rigidity  Scale/Flexibility  Scale 

•  Tolerance  of  Ambiguity 

•  Cognitive  Reflection  Test 

Of  these  assessments,  only  the  CCTDI  is  commercially  available.  The  remaining  assessments 
were  developed  by  individual  researchers  for  use  in  specific  experiments.  Some  of  the 


79  Furnham  &  Ribchester,  p.  193  . 
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researchers  included  the  actual  assessment  in  their  reports;  other  assessments  were  only 
described.  Since  these  assessments  were  developed  primarily  for  use  in  experiments,  many  of 
the  assessments  evolved  over  time.  Additionally,  for  some  thinking  dispositions,  different 
researchers  developed  their  own  assessments  so  that  one  thinking  disposition  might  be  assessed 
in  multiple  ways.  Therefore  it  was  sometimes  difficult  to  find  a  definitive  version  of  an 
assessment. 

In  this  section  of  the  report  I  provide: 

•  A  brief  description  of  each  assessment 

•  Sample  questions  when  samples  were  available 

•  Evaluations  of  the  assessment  in  order  to  provide  a  non-biased  view  of  the 
assessment  usefulness.  Note  that  I  was  not  always  able  to  find  independent 
evaluations  for  all  assessments. 

4.3.1.  CCTDI 

Peter  Facione  of  Insight  Assessments  was  the  primary  developer  of  the  CCTDI.  As  the  only 
commercially  available  thinking  dispositions  assessment,  far  more  has  been  written  about  this 
assessment  than  any  other  assessment.  Because  it  is  a  commercial  product,  Facione  has  made 
every  effort  to  demonstrate  its  reliability,  validity,  and  usefulness.80  Additionally,  the  CCTDI  is 
a  broad  assessment  whose  goal  is  to  provide  a  holistic  evaluation  of  all  of  the  dispositions 
believed  to  be  needed  by  an  effective  critical  thinker. 

The  CCTDI  grew  out  of  the  APA  study  that  created  the  consensus  definition  of  critical  thinking. 
As  one  component  of  their  work,  the  participants  defined  the  characteristics  of  the  ideal  critical 
thinker: 

The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  open- 
minded,  flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  in 
making  judgments,  willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters, 
diligent  in  seeking  relevant  information,  reasonable  in  the  selection  of  criteria,  focused  in 

inquiry,  and  persistent  in  seeking  results  which  are  as  precise  as  the  subject  and  the 

8 1 

circumstances  of  inquiry  '  permit. 

Facione  explained  why  this  was  a  good  starting  point  for  developing  critical  thinking 
assessments: 

The  process  of  developing  a  good  educational  assessment  tool  of  any  kind  begins  with 
the  construct,  or  idea,  that  one  seeks  to  measure.  The  construct-validity  of  the  instrument 
depends  on  how  well  that  idea  has  been  articulated  and  how  well  the  tool  captures  that 
idea.  The  Delphi  Report  provided  a  wonderful  opportunity  for  tool  development,  since  it 


80  See  Facione,  P.A.  &  Facione,  N.C.  California  Critical  Thinking  Disposition  Inventory’  (CCTDI)  Test  Manual, 
California  Academic  Press,  Millbrae,  CA,  2010. 

81  Facione,  P.A.  Critical  Thinking:  A  Statement  of  Expert  Consensus  for  Purposes  of  Educational  Assessment  and 
Instruction.  Research  Findings  and  Recommendations,  American  Philosophical  Association,  Newark,  DE,  1990,  p. 
3. 
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expressed  a  consensus  construct  of  CT.  The  conceptualization  used  to  structure  an 
instrument,  thus,  was  much  more  than  simply  a  single  individual’s  notion  of  CT.82 

The  CCTDI  evaluates  the  strength  of  an  individual’s  disposition  toward: 

Truth-Seeking:  Truth-seeking  is  the  habit  of  always  desiring  the  best  possible 
understanding  of  any  given  situation;  it  is  following  reasons  and  evidence  wherever  they 
may  lead,  even  if  they  lead  one  to  question  cherished  beliefs.  Truth-seekers  ask  hard, 
sometimes  even  frightening  questions;  they  do  not  ignore  relevant  details;  they  strive  not 
to  let  bias  or  preconception  color  their  search  for  knowledge  and  truth.  The  opposite  of 
truth-seeking  is  bias  which  ignores  good  reasons  and  relevant  evidence  in  order  not  to 
have  to  face  difficult  ideas. 

Open-Mindedness:  Open-mindedness  is  the  tendency  to  allow  others  to  voice  views 
with  which  one  may  not  agree.  Open-minded  people  act  with  tolerance  toward  the 
opinions  of  others,  knowing  that  often  we  all  hold  beliefs  which  make  sense  only  from 
our  own  perspectives.  Open-mindedness,  as  used  here,  is  important  for  harmony  in  a 
pluralistic  and  complex  society  where  people  approach  issues  from  different  religious, 
political,  social,  family,  cultural,  and  personal  backgrounds.  The  opposite  of  open- 
mindedness  is  closed-mindedness  and  intolerance  for  the  ideas  of  others. 

Analyticity  also  describable  as  Foresightfulness:  Analyticity  is  the  tendency  to  be  alert 
to  what  happens  next.  This  is  the  habit  of  striving  to  anticipate  both  the  good  and  the  bad 
potential  consequences  or  outcomes  of  situations,  choices,  proposals,  and  plans.  The 
opposite  of  analyticity  is  being  heedless  of  consequences,  not  attending  to  what  happens 
next  when  one  makes  choices  or  accepts  ideas  uncritically. 

Systematicity:  Systematicity  is  the  tendency  or  habit  of  striving  to  approach  problems  in 
a  disciplined,  orderly,  and  systematic  way.  The  habit  of  being  disorganized  is  the 
opposite  characteristic  to  systematicity.  The  person  who  is  strong  in  systematicity  may  or 
may  not  actually  know  or  use  a  given  strategy  or  any  particular  pattern  in  problem 
solving,  but  they  have  the  mental  desire  and  tendency  to  approach  questions  and  issues  in 
such  an  organized  way. 

Critical  Thinking  Self-Confidence:  The  tendency  to  trust  the  use  of  reason  and 
reflective  thinking  to  solve  problems  is  reasoning  self-confidence.  This  habit  can  apply 
to  individuals  or  to  groups;  as  can  the  other  dispositional  characteristics  measured  by  the 
CCTDI.  We  as  a  family,  team,  office,  community,  or  society  can  have  the  habit  of  being 
trustful  of  reasoned  judgment  as  the  means  of  solving  our  problems  and  reaching  our 
goals.  The  opposite  is  the  tendency  to  be  mistrustful  of  reason,  to  consistently  devalue  or 
be  hostile  to  the  use  of  careful  reason  and  reflection  as  a  means  to  solving  problems  or 
discovering  what  to  do  or  what  to  believe. 

Inquisitiveness:  Inquisitiveness  is  intellectual  curiosity.  It  is  the  tendency  to  want  to 
know  things,  even  if  they  are  not  immediately  or  obviously  useful  at  the  moment.  It  is 
being  curious  and  eager  to  acquire  new  knowledge  and  to  learn  the  explanations  for 
things  even  when  the  applications  of  that  new  learning  are  not  immediately  apparent. 

The  opposite  of  inquisitiveness  is  indifference. 


^  Facione,  P.A.,  Facione,  N.C.  &  Giancarlo,  C.A.  "The  Disposition  Toward  Critical  Thinking:  Its  Character, 
Measurement,  and  Relationship  to  Critical  Thinking  Skill,"  Informal  Logic,  Vol.  20,  No.  I,  2000,  p.  12. 
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Judiciousness  also  called  Maturity  of  Judgment:  Cognitive  maturity  is  the  tendency  to 
see  problems  as  complex,  rather  than  black  and  white.  It  is  the  habit  of  making  a 
judgment  in  a  timely  way,  not  prematurely,  and  not  with  undue  delay.  It  is  the  tendency 
of  standing  firm  in  one's  judgment  when  there  is  reason  to  do  so,  but  changing  one's  mind 
when  that  is  the  appropriate  thing  to  do.  It  is  prudence  in  making,  suspending,  or  revising 
judgment.  It  is  being  aware  that  multiple  solutions  may  be  acceptable  while  appreciating 
the  need  to  reach  closure  in  certain  circumstances  even  in  the  absence  of  complete 
knowledge.  The  opposite,  cognitive  immaturity,  is  characterized  by  being  imprudent, 
black  and-white  thinking,  failing  to  come  to  closure  in  a  timely  way,  stubbornly  refusing 
to  change  one's  mind  when  reasons  and  evidence  would  indicate  one  is  mistaken,  or 
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foolishly  revising  one's  opinions  willy-nilly  without  substantial  reason  for  doing  so. 

Since  the  CCTDI  is  a  commercial  product,  Insight  Assessments  protects  the  contents  and  has  not 
published  a  complete  list  of  the  questions  it  contains.  However,  Facione  did  include  some 
sample  questions  in  a  1995  article: 

Truthseeking 

"Everyone  always  argues  from  their  own  self-interest,  including  me." 

"If  there  are  four  reasons  in  favor  and  one  against,  I'll  go  with  the  four."84 
Openmindedness: 

"It's  important  to  me  to  understand  what  other  people  think  about  things." 

85 

"It  concerns  me  that  I  might  have  biases  of  which  I  am  not  aware." 

Analytic  ity 

"If  bothers  me  when  people  rely  on  weak  arguments  to  defend  good  ideas." 

"People  need  reasons  if  they  are  going  to  disagree  with  another's  opinion."86 
Systematicity 

"I  always  focus  on  the  question  before  I  attempt  to  answer  it."87 
CT  Self-Confidence: 

"Tests  that  require  thinking,  not  just  memorization,  are  better  for  me." 

"I  take  pride  in  my  ability  to  understand  the  opinions  of  others."88 
Inquisitiveness 

"No  matter  what  the  topic,  I  am  eager  to  know  more  about  it," 

89 

"Learn  everything  you  can,  you  never  know  when  it  could  come  in  handy." 


83  Facione,  P.A.  &  Facione,  N.C.  California  Critical  Thinking  Disposition  Inventory’  (CCTDI)  Test  Manual, 
California  Academic  Press,  Millbrae,  CA,  2010,  pp.  9-10. 

84  Facione,  P.A.,  Sanchez,  C.  &  Facione,  N.C.  "The  Disposition  Toward  Critical  Thinking,"  The  Journal  of  General 
Education,  Vol.  44,  No.  1,  1995,  p.  8. 

85  Facione  et  al.,  pp.  6-7. 

86  Facione  et  al.,  p.  8. 

87  Facione  et  al.,  p.  7. 

88  Facione  et  al.,  p.  9. 
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Maturity  of  Judgment: 

"The  best  argument  for  an  idea  is  how  you  feel  about  it  at  the  moment." 

"Things  are  as  they  appear  to  be."90 

What  Others  Have  Written  About  the  CCTDL.  The  Buros  Institute  of  Mental  Measurement 
provides  an  evaluation  of  every  commercially-available  cognitively-oriented  assessment.  Buros 
provides  one  or  two  separate  evaluations  for  each  assessment  selected  for  review.  Two 
reviewers  of  the  CCTDI  expressed  several  concerns,  which  are  included  below. 

Carolyn  M.  Callahan,  Professor,  Curry  School  of  Education,  University  of  Virginia,  wrote: 

The  manual  is  clearly  written  and  the  instructions  for  administration  and  scoring  of  the 
test  are  easily  followed.  Further,  the  process  of  translating  the  point  totals  into  standard 
scores  is  easily  executed.  However,  the  interpretation  of  the  resulting  standard  scores  is 
questionable.  These  scores  are  labeled  T-scores  in  footnotes  which  describe  their 
derivation,  yet  they  are  not  derived  from  z-scores  or  normative  groups.  If  the  scores  are 
presented  as  T-scores,  there  is  a  danger  of  misinterpretation  of  the  scores  as 
representative  of  standing  in  some  normative  group.  Further,  the  authors  have  presented 
"cut  off  scores"  based  on  a  theoretical  rationale  for  a  positive  or  negative  disposition  for 
critical  thinking,  but  present  no  empirical  evidence  that  individuals  scoring  above  or 
below  those  cutoff  scores  will  be  better  or  worse  critical  thinkers.  No  normative  data  are 
provided  in  the  manual  and  data  comparing  the  percentage  of  undergraduate  and  graduate 
students  scoring  above  and  below  the  cut  scores  are  highly  questionable  given  the  small 
number  of  post-baccalaureate  cases  (34). 91 

Salvador  Hector  Ochoa,  Assistant  Professor  of  Educational  Psychology  at  Texas  A&M 
University  wrote: 

The  content  validity  of  the  test  is  based  on  claims  the  items  are  derived  from  the 
consensus  of  46  theoreticians  regarding  the  dispositional  dimension  of  critical  thinking. 
An  original  set  of  150  statements  was  reduced  to  75  based  on  the  psychometric 
performance  of  the  items  on  a  pilot  version  of  the  instrument.  Although  the  subscales  are 
described  in  depth,  little  information  is  provided  on  how  the  characteristics  list  was 
translated  into  the  statements  on  the  instrument  itself  or  how  these  were  reviewed  for 
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accurate  reflection  of  the  characteristic. 

A  number  of  studies  have  also  suggested  that  there  are  four  primary  thinking  dispositions  rather 
than  the  seven  reported  by  Facione.  For  example,  Walsh  et  al.  found  only  four  thinking 
dispositions  through  a  principal  component  analysis  process  similar  to  Facione’s: 


89  Facione  et  al.,  p.  6. 

90  Facione  et  al.,  p.  9. 

9ICallahan,  C.M.  "Test  Review  of  the  California  Critical  Thinking  Dispositions  Inventory,"  in  The  Twelfth  Mental 
Measurements  Yearbook ,  eds.  J.C.  Conoley  &  J.C.  Impara,  Retrieved  from  the  Buros  Institute’s  Test  Reviews  Online 
website:  http://www.unl.edu/buros,  1995. 

92  Ochoa,  S.H.  "Test  Review  of  the  California  Critical  Thinking  Dispositions  Inventory,"  in  The  Twelfth  Mental 
Measurements  Yearbook,  eds.  J.C.  Conoley  &  J.C.  Impara,  Retrieved  from  the  Buros  Institute’s  Test  Reviews  Online 
website:  http://www.unl.edu/buros,  1995. 
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•  Intellectual  prowess 

•  Objectivity 

•  Systemacity 

•  •  93 

•  Receptivity 

4.3.2.  AOT  Scale 

The  AOT  Scale  was  developed  by  Keith  Stanovich  and  Richard  West  to  support  their 
experiments  on  the  impact  thinking  dispositions  have  on  cognitive  performance.  As  noted  in  the 
previous  section  of  this  report,  Stanovich  and  West  combined  several  subscales  to  create  the 
overall  AOT  Scale. 

Stanovich  and  West  have  modified  the  assessment  over  time.  These  sample  questions  came 
from  the  2003  version  of  the  AOT  Scale: 

1 .  Even  though  freedom  of  speech  for  all  groups  is  a  worthwhile  goal,  it  is 
unfortunately  necessary  to  restrict  the  freedom  of  certain  political  groups. 

2.  What  beliefs  you  hold  have  more  to  do  with  your  own  personal  character  than  the 
experiences  that  may  have  given  rise  to  them. 

3.  I  tend  to  classify  people  as  either  for  me  or  against  me. 

4.  A  person  should  always  consider  new  possibilities. 

5.  There  are  two  kinds  of  people  in  this  world:  those  who  are  for  the  truth  and  those 
who  are  against  the  truth. 

6.  Changing  your  mind  is  a  sign  of  weakness.94 

What  Others  Have  Written  About  the  AOT  Scale:  Marsh  and  Pastor  of  James  Madison 
University  provided  this  evaluation  of  the  AOT  Scale: 

The  purpose  of  the  current  study  was  to  examine  the  emergent  factor  structure  of 
Stanovich  and  West’s  (2007)  AOT  scale  in  an  effort  to  determine  whether  scores  on  the 
AOT  accurately  represent  actively  open-minded  thinking  as  conceptualized  by  Stanovich 
and  West  (1997,  2007).  Stanovich  and  West  (2007)  suggest  calculating  an  AOT  scale 
total  score  by  summing  participant  responses  across  item  statements  originally  developed 
to  address  six  facets  of  the  actively  open-minded  thinking  construct,  including,  dogmatic 
thinking,  categorical  thinking,  flexible  thinking,  openness  to  values,  belief  identification, 
and  counterfactual  thinking.  The  suggested  calculation  of  an  AOT  total  score  implies 
that  Stanovich  and  West  believe  the  scale  to  be  unidimensional.  However,  this  does  not 
appear  to  be  the  case  based  upon  the  findings  of  the  current  study.  AOT  items  did  not 
load  on  the  six  subscales  originally  developed  by  Stanovich  and  West  (2007)  nor  were 
they  unidimensional;  instead,  a  4-factor  solution  emerged. 

EFA  results  not  only  indicated  that  calculation  of  an  AOT  total  score  is  inappropriate,  but 
that  the  current  version  of  the  Stanovich  and  West  (2007)  AOT  scale  needs  to  be  revised  and 
conceptualization  of  the  actively  open-minded  thinking  construct  revisited.  Examination  of 


93  Walsh,  C.M.,  Seldomridge,  L.A.  &  Badros,  K.K.  "California  Critical  Thinking  Disposition  Inventory:  Further 
Factor  Analytic  Examination,"  Perceptual  &  Motor  Skills,  Vol.  104,  No.  1,  2007,  p.  148. 

94  Sa,  W.,  Kelley,  C„  Flo,  C.,  Stanovich,  K.E.  &  West,  R.F.  Composite  Actively  Open-Minded  Thinking  Scale  — 
11/6/03,  2003,  pp.  1-2. 
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the  AOT  inter-item  correlation  matrix  revealed  that  inter-item  correlations  were  quite  low, 
indicating  that  the  4 1  items  currently  on  the  AOT  scale  may  be  measuring  distinct,  or 
seemingly  unrelated  concepts.  Given  the  low  inter- item  correlations,  it  is  not  surprising  that 
although  the  EFA  resulted  in  an  interpretable  4-factor  solution,  there  were  a  series  of 
problematic  items  identified  in  need  of  substantial  revision  or  removal  (e.g.,  items  that  failed 
to  load  across  factor  solutions,  items  that  failed  to  load  on  any  of  the  emergent  four  factors). 
In  addition,  the  emergent  factors  failed  to  adequately  represent  theorized  aspects  of  actively 
open-minded  thinking  (e.g.,  factor  3  which  did  not  appear  to  represent  a  theorized  aspect  of 
actively  open-minded  thinking,  factor  4  which  only  consisted  of  two  items).  Moreover, 
estimates  regarding  the  percentage  of  variance  accounted  for  in  AOT  scores  by  each  of  the 
four  factors,  item  communalities,  and  total  variance  accounted  for  in  AOT  scores  by  all  four 
factors  further  support  the  distinct  nature  of  AOT  item  statements.95 

4.3.3.  Thinking  Dispositions  Questionnaire 

Stanovich  and  West  report  that  they  created  an  assessment  they  called  the  Thinking  Dispositions 
Questionnaire.  This  assessment  incorporates  multiple  assessments  addressing  individual 
characteristics  into  one  composite  assessment.  Components  of  the  Thinking  Dispositions 
Questionnaire  include: 

•  Flexible  Thinking 

•  Openness-Ideas 

•  Openness- Values 

•  Absolutism 

•  Dogmatism 

•  Categorical  Thinking 

•  Superstitious  Thinking 

•  Counterfactual  Thinking 

•  Outcome  Bias 

•  Social  Desirability  Response  Bias96 

This  assessment  appears  to  have  significant  overlap  with  the  AOT  Scale.  In  fact,  when  I  asked 
Dr.  Stanovich  for  a  copy  of  the  Thinking  Dispositions  Questionnaire,  he  sent  a  copy  of  the  AOT 
Scale. 

What  Others  Have  Written  About  the  Thinking  Dispositions  Questionnaire-.  I  found  no 
independent  assessments  of  the  Thinking  Dispositions  Questionnaire. 

4.3.4.  NFC  Scale 

John  Cacioppo  and  Richard  Petty  developed  the  NFC  Scale  and  reported  how  they  developed  the 
scale  in  a  1982  article  entitled  “The  Need  for  Cognition.”  The  article  abstract  provided  a  useful 
summary  of  their  work: 


95  Marsh,  K.R.  &  Pastor,  D.A.  "Stanovich  and  West’s  (2007)  Actively  Open-Minded  Thinking  Scale:  An 
Examination  of  Factor  Structure,"  Annual  Meeting  of  the  Association  for  Psychological  Science,  Washington  D.C., 
May  26-29,  2011,  pp.  11-12. 

96  Stanovich,  K.E.  &  West,  R.F.  "Reasoning  Independently  of  Prior  Belief  and  Individual  Differences  in  Actively 
Open-Minded  Thinking,"  Journal  of  Educational  Psychology’,  Vol.  89,  No.  2,  1997,  pp.  346-347. 
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Four  studies  are  reported  in  which  a  scale  to  assess  the  need  for  cognition  (i.e.,  the 
tendency  for  an  individual  to  engage  in  and  enjoy  thinking)  was  developed  and  validated. 
In  Study  1  a  pool  of  items  was  administered  to  groups  known  to  differ  in  need  for 
cognition.  Members  of  a  university  faculty  served  as  subjects  in  the  high-need-for- 
cognition  group,  whereas  assembly  line  workers  served  as  subjects  in  the  low-need-for- 
cognition  group.  The  criteria  of  ambiguity,  irrelevance,  and  internal  consistency  were 
used  to  select  the  items  for  subsequent  studies.  A  factor  analysis  was  performed  on  the 
selected  items  and  yielded  one  major  factor.  In  study  2  the  scale  was  administered  to  a 
more  homogeneous  population  (400  undergraduates)  to  validate  the  factor  structure 
obtained  in  Study  1  and  to  determine  whether  the  scale  tapped  a  construct  distinct  from 
test  anxiety  and  cognitive  style.  The  factor  structure  was  replicated  in  Study  2,  responses 
to  the  need  for  cognition  scale  were  predictably  and  weakly  related  to  cognitive  style,  and 
responses  were  unrelated  to  test  anxiety.  In  Study  3,  104  subjects  completed  need  for 
cognition,  social  desirability,  and  dogmatism  scales  and  indicated  what  their  American 
College  Test  scores  were.  Results  indicated  that  need  for  cognition  was  related  weakly 
and  negatively  to  being  close  minded,  unrelated  to  social  desirability,  and  positively 
correlated  with  general  intelligence.  Study  4  replicated  the  major  findings  of  Study  3  and 
furnished  evidence  of  the  predictive  validity  of  the  Need  for  Cognition  Scale:  Attitudes 
toward  simple  and  complex  versions  of  a  cognitive  task  appeared  indistinguishable  until 
the  subjects'  need-  for  'cognition  was  considered.  The  theoretical  utility  of  the  construct 
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and  measure  of  need  for  cognition  are  discussed. 


The  scale  initially  contained  34  questions.  Cacioppo  and  Petty  later  condensed  it  to  18 
questions.  Sample  questions  include: 


1 .  I  would  prefer  complex  to  simple  problems. 

2.  1  like  to  have  the  responsibility  of  handling  a  situation  that  requires  a  lot  of 
thinking. 

3.  Thinking  is  not  my  idea  of  fun. 

4.  I  would  rather  do  something  that  requires  little  thought  than  something  that  is  sure 
to  challenge  my  thinking  abilities. 

5.  I  try  to  anticipate  and  avoid  situations  where  there  is  likely  chance  I  will  have  to 
think  in  depth  about  something. 

QO 

6.  I  find  satisfaction  in  deliberating  hard  and  for  long  hours. 


What  Others  Have  Written  About  the  NFC  Scale :  Ann  Bost  of  Wabash  University  provided  a 
brief  evaluation  of  the  NFC  Scale: 


Based  on  previous  research,  the  Need  for  Cognition  Scale  appears  to  be  a  valid  and 
reliable  measure  of  individuals’  tendencies  to  pursue  and  enjoy  the  process  of  thinking — 
that  is,  of  their  "need  for  cognition"  (Cacioppo  &  Petty,  1982;  Cacioppo,  Petty,  Feinstein, 
&  Jarvis,  1996;  Cacioppo  et  al.,  1984;  Sadowski,  1993;  Sadowski  &  Gulgoz,  1992b). 
Need  for  Cognition  scores  are  not  influenced  by  whether  an  individual  is  male  or  female, 


97  Cacioppo,  J.T.  &  Petty,  R.E.  "The  Need  for  Cognition,"  Journal  of  Personality  and  Social  Psychology ,  Vol.  42, 
No.  1,  1982,  p.  116. 

98  Cacioppo,  J.T.,  Petty,  R.E.  &  Kao,  C.F.  "The  Efficient  Assessment  of  Need  for  Cognition,"  Journal  of  Personality 
Assessment,  Vol.  48,  No.  3,  1984,  p.  307. 
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or  by  differences  in  the  individual’s  level  of  test-taking  anxiety  or  cognitive  style  (the 
particular  way  that  an  individual  accumulates  and  merges  information  during  the  thinking 
process).  In  general,  scores  on  the  Need  for  Cognition  Scale  also  are  not  impacted  by 
whether  or  not  the  individuals  are  trying  to  paint  a  favorable  picture  of  themselves 
(Cacioppo  &  Petty,  1982)." 

4.3.5.  Need  for  Cognitive  Closure  Scale  (NFCS) 

Webster  and  Kruglanski  developed  the  NFCS  and  described  its  development  in  “Individual 
Differences  in  Need  for  Cognitive  Closure.” 

In  developing  the  NFCS,  our  initial  item-generation  process  attempted  to  capture  a  broad 
sense  of  the  construct.  In  accordance  with  the  underlying  theory  (Kruglanski,  1989, 

1990a  1990b),  we  reasoned  that  the  need  for  closure  may  express  itself  in  various  ways. 
Thus,  we  treated  it  as  a  latent  variable  manifest  through  different  aspects  (Carver,  1989). 
In  particular,  we  identified  five  major  such  aspects  assumed  to  broadly  represent  the 
universe  of  the  construct  and  generated  diverse  items  correspondent  with  those  aspects.100 

The  five  major  aspects  of  the  need  for  cognitive  closure  are:  preference  for  order  and  structure, 
discomfort  with  ambiguity,  decisiveness,  predictability  and  close-mindedness.  Sample  questions 
from  the  NFCS  that  measure  these  characteristics  include: 

1 .  I  think  that  having  clear  rules  and  order  at  work  is  essential  for  success. 

2.  I'd  rather  know  bad  news  than  stay  in  a  state  of  uncertainty. 

3.  I  usually  make  important  decisions  quickly  and  confidently. 

4.  I  don't  like  to  go  into  a  situation  without  knowing  what  I  can  expect  from  it. 

5.  I  do  not  usually  consult  many  different  opinions  before  forming  my  own  view.101 

What  Others  Have  Written  About  the  NFCS:  Although  Kruglanski  and  Webster  included 
reliability  and  validity  information  in  their  article,  I  found  no  independent  evaluations  of  the 
reliability  and  validity  of  the  NFCS. 

4.3.6.  Dogmatism  Scale 

Milton  Rokeach  developed  the  Dogmatism  Scale  and  described  his  approach  to  the  assessment  in 
The  Open  and  Closed  Mind : 

Our  assumption  was  that  if  a  person  strongly  agrees  with  such  statements  it  would 
indicate  that  he  possesses  one  extreme  of  the  particular  characteristic  being  tapped,  and  if 
he  strongly  disagrees,  that  he  possesses  the  opposite  extreme.  Insofar  as  possible,  we 
looked  for  statements  that  express  ideas  familiar  to  the  average  person  in  his  everyday 
life.  Some  of  the  statements  appearing  in  the  Dogmatism  Scale  were  inspired  by 
spontaneous  remarks  we  overheard  being  made  by  persons  we  thought  intuitively  to  be 
closed-minded.  Above  all,  each  statement  in  the  scale  had  to  be  designed  to  transcend 
specific  ideological  positions  in  order  to  penetrate  to  the  formal  and  structural 


99  Bost,  A.  The  Need  for  Cognition  Scale,  Wabash  College.  Available:  http://www.liberalarts.wabash.edu/ncs/. 
Viewed:  July  12,  2011. 

100  Webster,  D.M.  &  Kruglanski,  A.  W.  "Individual  Differences  in  Need  for  Cognitive  Closure,"  Journal  of 
Personality  and  Social  Psychology’,  Vol.  67,  No.  6,  1994,  p.  1050. 

101  Webster  &  Kruglanski,  p.  1050. 
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characteristics  of  all  positions.  Persons  adhering  dogmatically  to  such  diverse  viewpoints 
as  capitalism  and  communism,  Catholicism  and  anti-Catholicism,  should  all  score 
together  at  one  end  of  the  continuum,  and  should  all  score  in  a  direction  opposite  to 
others  having  equally  diverse  yet  undogmatic  viewpoints.  Thus,  it  was  our  hope  that  the 

Dogmatism  Scale  could  be  employed  as  a  research  tool  not  only  in  the  Western  countries 

102 

but  also  in  the  Soviet  Union  and  in  other  Eastern  countries. 

The  final  version  of  the  Dogmatism  Scale  contained  40  questions.  The  following  are  sample 
questions  from  the  assessment: 

1 .  Even  though  freedom  of  speech  for  all  groups  is  a  worthwhile  goal,  it  is 
unfortunately  necessary  to  restrict  the  freedom  of  certain  political  groups. 

2.  While  the  use  of  force  is  wrong  by  and  large,  it  is  sometimes  the  only  way  possible 
to  advance  a  noble  ideal. 

3.  It  is  only  natural  that  a  person  would  have  a  much  better  acquaintance  with  ideas  he 
believes  in  than  with  ideas  he  opposes. 

4.  Fundamentally,  the  world  we  live  in  is  a  pretty  lonesome  place. 

5.  In  a  heated  discussion  I  generally  become  so  absorbed  in  what  I  am  going  to  say 
that  I  forget  to  listen  to  what  the  others  are  saying. 

1  AO 

6.  It's  all  too  true  that  people  just  won’t  practice  what  they  preach. 

A  number  of  researchers  have  updated  the  Dogmatism  Scale  to  shorten  it,  to  make  the  questions 
it  uses  more  balanced,  and  to  make  the  questions  more  pertinent  to  modern  audiences.  For 
example,  Troldahl  and  Powell  developed  a  shorter  version  in  1965: 

Rokeach  recently  introduced  a  theory  about  different  styles  of  belief  systems  that  people 
have.  The  theory  is  an  outgrowth  of  work  on  the  Authoritarian  Personality.  The  theory 
has  not  been  utilized  much  in  field  studies  because  the  primary  measuring  instrument,  the 
"dogmatism"  scale,  contains  40  items.  In  this  article,  the  authors  recommend  shorter 
versions  of  the  dogmatism  scale,  indicating  which  items  should  be  used  to  maintain  the 
reliability  of  the  measuring  instrument.  The  short  fonns  were  developed  from  data 
collected  in  two  field  studies.104 

The  interest  in  developing  a  useful  Dogmatism  Scale  continues  to  the  present  time.  Shearman 
and  Levine  proposed  a  new  version  of  the  scale  in  2006: 

Therefore,  based  on  the  observations  noted,  this  study  attempts  to  update  the  dogmatism 
scale.  This  study  aims  to  clarify  the  conceptualization  of  dogmatism  as  a  closed 
cognitive  style  separate  from  political  connotations,  update  the  scale  for  practical  usage 


102  Rokeach,  M.  "The  Measurement  of  Open  and  Closed  Systems,"  in  The  Open  and  Closed  Mind,  ed.  M.  Rokeach, 
Basic  Books,  New  York,  NY,  1960,  p.  72. 

103  Rokeach,  pp.  74-76. 

104  Troldahl,  V.C.  &  Powell,  F.A.  "A  Short-Form  Dogmatism  Scale  for  Use  in  Field  Studies,"  Social  Forces,  Vol. 
44,  No.  2,  1965,  p.  211. 
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with  current  language  and  fewer  items,  examine  the  scale  as  a  unidimensional  construct, 
and  finally  conduct  validity  testing  for  the  revised  scale.105 

Sheannan  and  Levine  proposed  changes  to  the  scale  to  accomplish  the  following: 

The  present  intention  is  to  make  this  concept  of  the  closed  vs.  open  nature  of  cognitive 
style  clear  and  precise.  To  this  end,  four  central  characteristics  of  the  dogmatic  cognitive 
style  are  proposed:  the  degree  of  open-mindedness  verses  closed-mindedness,  the  extent 
that  an  individual  believes  in  a  single  correct  view,  the  extent  to  which  one  rejects  ideas 
or  viewpoints  that  are  disagreement  with  one's  own  opinion,  and  blind  respect  or 
excessive  reliance  on  authority. 106 

What  Others  Have  Said  About  the  Dogmatism  Scale:  Without  a  definitive  copy  of  the 
Dogmatism  Scale,  it  was  difficult  to  find  an  authoritative  independent  assessment.  I  did  find  a 
number  of  justifications  for  revising  the  Dogmatism  Scale.  For  example,  John  Ray  wrote: 

Since  the  publication  of  the  California  'F'  scale  (Adomo  et  ah,  1950)  two  major  criticisms 
have  been  levelled  against  it  as  a  measure  of  authoritarianism.  1 .  That  it  is  ideology 
specific  and  slanted  so  as  to  cause  conservatives  automatically  to  obtain  high  scores;  2. 
That  its  all-positive  wording  causes  acquiescence  and  authoritarianism  to  be  confounded. 

The  first  of  these  criticisms  was  largely  met  by  the  Rokeach  (1960)  dogmatism  scale  and 
Rokeach  (1967),  claims  that  his  scale  is  sufficiently  free  from  response  bias  to  render  a 
balanced  version  unnecessary.  Although  this  may  be  true  to  some  extent,  it  does  seem 
desirable  to  lay  the  doubts  about  "the  acquiescent  personality"  completely  to  rest  where- 
ever  it  is  possible  to  do  so.  The  voluminous  literature  on  the  one-way  wording  of  the  'F' 
scale  is  sufficient  to  inspire  in  every  user  of  scales  the  desire  to  preclude  from  the 
beginning  a  "response  set"  interpretation  of  his  own  results.107 

4.3.7.  Rigidity  Scale/Flexibility  Scale 

Gough  and  Sanford  developed  a  Rigidity  Scale  that  Rokeach  described  as  “widely  used”  in  his 
1960  book.  The  following  are  some  of  the  questions  included  in  the  assessment: 

1 .  I  always  put  on  and  take  off  my  clothes  in  the  same  order. 

2.  I  never  miss  going  to  church. 

3.  I  usually  check  more  than  once  to  be  sure  that  I  have  locked  a  door,  put  out  a  light, 
or  something  of  the  sort. 

4.  I  often  find  myself  thinking  of  the  same  tunes  or  phrases  for  days  at  a  time. 

108 

5.  I  prefer  to  stop  and  think  before  I  act  even  on  trifling  matters. 

Gough  later  developed  the  California  Psychological  Inventory  (CPI),  a  commercially-available 
personality  assessment.  He  included  the  concept  of  rigid/non  rigid  thinking  in  the  CPI  and 
labeled  the  trait  as  Flexibility.  He  describes  the  polar  endpoints  of  the  Flexibility  Scale  as: 


105  Sheannan,  S.M.  &  Levine,  T.R.  "Dogmatism  Updated:  A  Scale  Revision  and  Validation,"  Communication 
Quarterly,  Vol.  54,  No.  3,  2006,  p.  379. 

106  Shearman  &  Levine,  p.  379. 

107  Ray,  J.J.  "The  Development  and  Validation  of  a  Balanced  Dogmatism  Scale,"  Australian  Journal  of  Psychology’, 
Vol.  22,  No.  3,  1970,  pp.  1-2. 

108  Rokeach,  M.,  McGovney,  W.C.  &  Denny,  M.R.  "Dogmatic  Thinking  Versus  Rigid  Thinking:  An  Experimental 
Distinction,"  in  The  Open  and  Closed  Mind,  ed.  M.  Rokeach,  Basic  Books,  New  York,  NY,  1960,  pp.  184-185. 
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Low  Flexibility: 

Prefers  predictability  and  consistency,  uncomfortable  with  ambiguity,  programmed  and 
planful,  well-organized.109 

High  Flexibility: 

Likes  change  and  variety,  finds  ordinary  routine  boring,  quick- thinking  and  clever.1 10 

Stanovich  and  West  also  developed  a  Flexible  Thinking  scale  as  part  of  their  Thinking 
Dispositions  Questionnaire: 

Flexible  Thinking  Scale.  We  devised  the  items  on  the  Flexible  Thinking  scale.  The  design 
of  the  items  was  influenced  by  a  variety  of  sources  from  the  critical  thinking  literature 
(e.g.,  Ennis,  1987;  Facione,  1992;  Nickerson,  1987;  Norris  &  Ennis,  1989;  Perkins  et  al., 
1993;  Zechmeister  &  Johnson,  1992)  but  most  specifically  by  the  work  of  Baron  (1985, 
1988,  1993).  Baron  has  emphasized  that  actively  open-minded  thinking  is  a  multifaceted 
construct  encompassing  the  cultivation  of  reflectiveness  rather  than  impulsivity,  the 
seeking  and  processing  of  information  that  disconfirms  one's  belief  (as  opposed  to 
confirmation  bias  in  evidence  seeking),  and  the  willingness  to  change  one's  beliefs  in  the 
face  of  contradictory  evidence.  There  were  10  items  on  the  Flexible  Thinking  scale,111 

The  following  are  sample  questions  from  the  Flexible  Thinking  scale: 

1 .  "If  I  think  longer  about  a  problem  I  will  be  more  likely  to  solve  it;" 

2.  "Intuition  is  the  best  guide  in  making  decisions;" 

3.  "People  should  always  take  into  consideration  evidence  that  goes  against  their 
beliefs” 

1 12 

4.  "There  is  nothing  wrong  with  being  undecided  about  many  issues," 

What  Others  Have  Written  About  the  Rigidity  or  Flexible  Thinking  Scales:  I  found  no 
independent  evaluations  of  the  reliability  and  validity  of  the  independent  Rigidity  or  Flexible 
Thinking  assessments. 

4.3.8  Tolerance/Intolerance  of  Ambiguity  Assessments 

The  most  widely  referenced  ambiguity  tolerance  assessment  was  developed  by  Budner  in  1962. 
The  assessment  is  known  as  the  Tolerance  of  Ambiguity  Scale  and  consists  of  16  questions. 

Some  sample  questions  from  this  assessment  include: 

1 .  An  expert  who  doesn’t  come  up  with  a  definite  answer  probably  doesn’t  know  too 
much. 

2.  I  would  like  to  live  in  a  foreign  country  for  a  while. 

3.  There  is  really  no  such  thing  as  a  problem  that  can’t  be  solved. 

4.  People  who  fit  their  lives  to  a  schedule  probably  miss  most  of  the  joy  of  living. 


109  Gough,  H.G.  &  Bradley,  P.  CPI  260  Client  Feedback  Report  Example,  CPP,  Inc.,  Mountain  View,  CA,  2003,  p. 
8. 

110  Gough  &  Bradley,  p.  8. 

111  Stanovich,  K.E.  &  West,  R.F.  "Reasoning  Independently  of  Prior  Belief  and  Individual  Differences  in  Actively 
Open-Minded  Thinking,"  Journal  of  Educational  Psychology’,  Vol.  89,  No.  2,  1997,  p.  346. 

112  Stanovich  &  West,  p.  346. 
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5. 


A  good  job  is  one  where  what  is  to  be  done  and  how  it  is  to  be  done  are  always 
clear.113 


Assessments  for  tolerance  of  ambiguity  have  undergone  significant  revisions  over  time.  Ryden 
and  Rosen  developed  a  new  assessment  in  1966.  McDonald  revised  this  assessment  in  1970  by 
adding  four  questions  to  it  to  create  what  he  called  the  AT-20.  Kirton  then  developed  a 
consolidated  assessment  from  Budner’s  and  McDonalds’s  work.  He  used  only  the  questions 
from  each  assessment  that  his  research  had  proved  to  be  the  most  useful  in  distinguishing 
between  tolerance  and  intolerance  for  ambiguity.  Sample  questions  from  Kirton’ s  assessment 
include: 

1 .  There's  a  right  way  and  a  wrong  way  to  do  almost  everything. 

2.  If  I  were  a  doctor,  I  would  prefer  the  uncertainties  of  a  psychiatrist  to  the  clear  and 
definite  work  of  someone  like  a  surgeon  or  x-ray  specialist, 

3.  I  don't  like  to  work  on  a  problem  unless  there  is  a  possibility  of  coming  out  with  a 
clear-cut  and  unambiguous  answer. 

4.  An  expert  who  doesn’t  come  up  with  a  definite  answer  probably  doesn’t  know  too 
much. 

5.  There  is  really  no  such  thing  as  a  problem  that  can’t  be  solved. 

6.  A  good  job  is  one  where  what  is  to  be  done  and  how  it  is  to  be  done  are  always 
clear. 

7.  What  we  are  used  to  is  always  preferable  to  what  is  unfamiliar. 1 14 

What  Others  Have  Written  About  Tolerance/Intolerance  of  Ambiguity  Assessments: 

McDonald  reported  positive  results  for  Budner’s  assessment: 

On  the  more  positive  side,  the  Budner  Scale  of  Tolerance-Intolerance  of  Ambiguity  has 
shown  fairly  good  internal  consistency  (Budner,  1962).  Reliabilities  (alpha  coefficients) 
reported  for  13  of  his  17  samples  ranged  from  .39  to  .62,  with  a  mean  of  approximately 
.49. 115 

Benjamin  et  al.  reported: 

Although  Budner’s  (1962)  scale  is  one  of  the  better  known  and  more  widely  used 
measures  of  tolerance  for  ambiguity  (Fumham,  1994),  reliability  estimates  for  the 
measure  tend  to  be  inconsistent.  When  Budner  tested  his  scale  on  17  different  samples,  he 
reported  alpha  reliabilities  ranging  from  .39  to  .62  (with  a  mean  alpha  of  .49),  indicating 
that  the  measure  was  low  in  internal  consistency.  More  recently,  Sobal  and  DeForge 
(1992)  reported  alphas  of  .63  and  .64  when  the  measure  was  administered  to  their  two 
samples.  In  addition,  Furnham  (1994)  found  that  Budner's  scale  had  a  lower  reliability 
estimate  (a  =  .59)  than  two  other  measures  of  tolerance  for  ambiguity.  Furnham  reported 


113  Budner,  S.  “Intolerance  of  Ambiguity  as  a  Personality  Variable,”  Journal  of  Personality ,  Vol.  30,  No.  1,  p.  34. 

114  Kirton,  M.J.  "A  Reanalysis  of  Two  Scales  of  Tolerance  of  Ambiguity,"  Journal  of  Personality  Assessment,  Vol. 
45,  No.  4,  1981,  p.  411. 

115  MacDonald,  A.P.  "Revised  Scale  for  Ambiguity  Tolerance:  Reliability  and  Validity,"  Psychological  Reports, 
Vol.  26,  No.  3,  1970,  p.  792. 
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an  alpha  of  .89  for  Norton's  (1975)  tolerance  for  ambiguity  scale,  and  an  alpha  of  .78  for 
Rydell  and  Rosen's  (1966;  see  also  MacDonald,  1970)  measure  of  ambiguity  tolerance.1 16 

Furnham  and  Ribchester  noted  challenges  with  the  concept  of  tolerance  of  ambiguity: 

Raphael,  Moss,  and  Cross  (1978)  offered  a  very  cautionary  note  that  unless  one  always 
partials  out  intelligence,  correlates  of  AT  may  be  illusory.  They  pointed  out  that  in  many 
studies  intelligence  is  not  considered,  but  where  it  is  shown  to  correlate  significantly  with 
AT,  it  is  rarely  covaried  out.  Indeed,  this  may  be  not  the  only  moderator  variable  of 
importance  in  research  in  this  area.  However,  this  very  piecemeal  research  has  been 
handicapped  particularly  because  of  the  way  AT  is  measured.  A  variety  of  questionnaire 
and  experimental  methods  have  been  used  to  classify  subjects,  and  many  have  dubious 
psychometric  validity.  Furthermore,  because  until  recently  few  concurrent  validation 
studies  were  done,  it  was  impossible  to  compare  results  from  different  studies  (Furnham, 
1994).  Thus,  whereas  the  vast  majority  of  one-off  studies  have  provided  modest 
confirmatory  results  for  their  AT-theory-derived  hypotheses,  the  field  has  not  really 
progressed  through  these  "correlated"  studies.117 

Several  researchers  noted  overlap  with  other  thinking  dispositions.  For  example,  McDonald 
noted: 

Additional  validational  evidence  was  reflected  in  the  significant  correlations  between  the 
AT-20  and  the  Rokeach  Dogmatism  Scale  (Fonn  E),  the  Gough-Sanford  Rigidity  Scale, 
and  the  F  Scale.118 

4.3.9.  Cognitive  Reflection  Test  (CRT) 

The  CRT  is  fundamentally  different  than  previously  reported  assessments.  All  assessments  so 
far  have  been  based  on  self-reporting,  with  the  possibility  that  subjects  could  respond  with  what 
they  believe  to  be  a  positive  answer.  The  CRT  asks  the  test  subject  to  perform  a  cognitive  task 
that  appears  to  be  simple  but  is  not.  This  task  is  designed  to  distinguish  between  people  who 
rush  to  quick  answers  from  those  who  are  willing  to  reflect  in  order  to  get  the  right  answer.  This 
task-oriented  assessment  of  willingness  to  reflect  before  providing  an  answer  appears  to  be  a 
valuable  addition  to  assessments  of  thinking  dispositions  that  rely  on  respondents’  self-reporting. 

The  developer  of  the  CRT,  Shane  Frederick,  explained  the  theory  underlying  his  assessment. 

Many  researchers  have  emphasized  the  distinction  between  two  types  of  cognitive 
processes:  those  executed  quickly  with  little  conscious  deliberation  and  those  that  are 
slower  and  more  reflective  (Epstein,  1994;  Sloman,  1996;  Chaiken  and  Trope,  1999; 
Kahneman  and  Frederick,  2002).  Stanovich  and  West  (2000)  called  these  “System  1”  and 
“System  2”  processes,  respectively.  System  1  processes  occur  spontaneously  and  do  not 
require  or  consume  much  attention.  Recognizing  that  the  face  of  the  person  entering  the 
classroom  belongs  to  your  math  teacher  involves  System  1  processes  -  it  occurs  instantly 
and  effortlessly  and  is  unaffected  by  intellect,  alertness,  motivation  or  the  difficulty  of  the 


116  Benjamin,  A.J.,  Riggio,  R.E.  &  Mayes,  B.T.  "Reliability  and  Factor  Structure  of  Budner's  Tolerance  for 
Ambiguity  Scale,"  Journal  of  Social  Behavior  and  Personality,  Vol.  11,  No.  3,  1996,  p.  626. 

117  Furnham,  A.  &  Ribchester,  T.  "Tolerance  of  Ambiguity:  A  Review  of  the  Concept,  its  Measurement  and 
Applications,"  Current  Psychology,  Vol.  14,  No.  3,  1995,  p.  184. 

118  MacDonald,  A.P.  "Revised  Scale  for  Ambiguity  Tolerance:  Reliability  and  Validity,"  Psychological  Reports, 
Vol.  26,  No.  3,  1970,  p.  797. 
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math  problem  being  attempted  at  the  time.  Conversely,  finding  Vl 9 1 63  to  two  decimal 
places  without  a  calculator  involves  System  2  processes  -  mental  operations  requiring 
effort,  motivation,  concentration,  and  the  execution  of  learned  rules.119 

By  contrast  with  the  square  root  example  above,  Frederick  proposed  simple  questions  that  have 
intuitive,  but  wrong,  answers.  He  proposed  that  these  simple  questions  will  distinguish  between 
individuals  who  are  willing  to  make  cognitive  effort  from  those  who  rush  to  the  obvious, 
intuitive  solution.  The  CRT  consists  of  three  questions: 

1 .  A  bat  and  a  ball  cost  $  1 . 10  in  total.  The  bat  costs  $  1 .00  more  than  the  ball. 

How  much  does  the  ball  cost? _ cents 

2.  If  it  takes  5  machines  5  minutes  to  make  5  widgets,  how  long  would  it  take  100 

machines  to  make  100  widgets? _ minutes 

3.  In  a  lake,  there  is  a  patch  of  lily  pads.  Every  day,  the  patch  doubles  in  size. 

If  it  takes  48  days  for  the  patch  to  cover  the  entire  lake,  how  long  would  it  take  for 
the  patch  to  cover  half  of  the  lake? _ days 

Frederick  compared  the  CRT  with  several  other  assessments  including  intelligence  measures 
such  as  the  SAT  and  the  dispositional  assessment  for  need  for  cognition. 

As  expected,  all  measures  correlate  positively  and  significantly  with  one  another.  The 

moderate  correlations  suggest  that  all  five  tests  likely  reflect  common  factors,  but  may 

also  measure  distinct  characteristics,  as  they  purport  to.  I  have  proposed  that  the  CRT 

measures  “cognitive  reflection” — the  ability  or  disposition  to  resist  reporting  the  response 

that  first  comes  to  mind.  The  NFC  is  advanced  as  a  measure  of  someone’s  “tendency  to 

engage  in  and  enjoy  thinking”  (Cacioppo  and  Petty,  1982),  but  relies  on  self-reports 

120 

rather  than  observed  behavior. 

Frederick  did  note  a  gender  related  difference  in  the  CRT : 

It  appears,  instead,  that  these  items  measure  something  that  men  have  more  of.  That 
something  may  be  mathematical  ability  or  interest,  since  the  CRT  items  have 
mathematical  content,  and  men  generally  score  higher  than  women  on  math  tests 
(Benbow  and  Stanley,  1980;  Halpern,  1986;  Hyde,  Fennema  and  Lamon,  1990;  Hedges 
and  Nowell,  1995).  However,  men  score  higher  than  women  on  the  CRT,  even 
controlling  for  SAT  math  scores.  Furthermore,  even  if  one  focuses  only  on  respondents 
who  gave  the  wrong  answers,  men  and  women  differ.  Women’s  mistakes  tend  to  be  of 
the  intuitive  variety,  whereas  men  make  a  wider  variety  of  errors.  For  example,  the 
women  who  miss  the  “widgets”  problem  nearly  always  give  the  erroneous  intuitive 
answer  “100,”  whereas  a  modest  fraction  of  the  men  give  unexpected  wrong  answers, 
such  as  “20”  or  “500”  or  “1.”  For  every  CRT  item  (and  several  other  similar  items  used 
in  a  longer  variant  of  the  test)  the  ratio  of  “intuitive”  mistakes  to  “other”  mistakes  is 

higher  for  women  than  for  men.  Thus,  the  data  suggest  that  men  are  more  likely  to 

121 

reflect  on  their  answers  and  less  inclined  to  go  with  their  intuitive  responses. 


119  Frederick,  S.  "Cognitive  Reflection  and  Decision  Making,"  Journal  of  Economic  Perspectives,  Vol.  19,  No.  4, 
2005,  p.  26. 

120  Frederick,  p.  34. 

121  Frederick,  p.  37. 

45 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 

88ABW-20 14-2455;  Cleared  21  May  2014 


What  Others  Have  Written  About  the  Cognitive  Reflection  Test :  After  years  of  working  with 
self  developed  thinking  disposition  assessments,  Stanovich  et  al.  used  the  CRT  in  their  studies  of 
individual  differences  in  reasoning  in  20 1 1 .  They  found  it  to  be  an  even  better  predictor  of 
cognitive  results  than  the  dispositional  assessments  they  had  been  using: 


The  CRT  (Frederick,  2005)  is  designed  to  measure  the  tendency  to  override  a  prepotent 

response  alternative  that  is  incorrect  and  to  engage  in  further  reflection  that  leads  to  the 

correct  response.  In  this  study,  we  showed  that  the  CRT  is  a  more  potent  predictor  of 

performance  on  a  wide  sample  of  tasks  from  the  heuristics-and-biases  literature  than 

measures  of  cognitive  ability,  thinking  dispositions,  and  executive  functioning.  Although 

the  CRT  has  a  substantial  correlation  with  cognitive  ability,  a  series  of  regression 

analyses  indicated  that  the  CRT  was  a  unique  predictor  of  performance  on  heuristics-and- 

biases  tasks.  It  accounted  for  substantial  additional  variance  after  the  other  measures  of 

individual  differences  had  been  statistically  controlled.  We  conjecture  that  this  is 

because  neither  intelligence  tests  nor  measures  of  executive  functioning  assess  the 

tendency  toward  miserly  processing  in  the  way  that  the  CRT  does.  We  argue  that  the 

CRT  is  a  particularly  potent  measure  of  the  tendency  toward  miserly  processing  because 

122 

it  is  a  performance  measure  rather  than  a  self-report  measure. 

4.3.10.  Thinking  Disposition  Assessments  and  Employee  Selection 

One  of  our  primary  research  questions  is  whether  it  is  feasible  and  worthwhile  to  use  thinking 
dispositions  as  one  part  of  intelligence  analyst  selection.  We  have  demonstrated  that  thinking 
dispositions  do  have  an  impact  on  cognitive  results,  thus  illustrating  the  potential  applicability  of 
thinking  dispositions.  Individuals  with  stronger  positive  thinking  dispositions  tend  to  have  better 
results  on  tasks  requiring  cognitive  effort.  We  have  also  found  a  number  of  different  assess¬ 
ments  that  can  determine  the  strength  of  a  particular  thinking  disposition,  thus  illustrating  that 
the  use  of  an  assessment  has  potential  feasibility.  We  now  turn  to  how  to  use  assessments  in 
employee  selection. 

CPP,  Inc.,  provider  of  the  CPI  mentioned  earlier  in  this  report,  developed  a  white  paper  on  how 
to  use  assessments  in  selecting  personnel.  They  recommended  the  following  steps  in  order  to 
develop  an  appropriate  selection  assessment: 

•  Conduct  a  job  analysis  to  identify  the  personal  characteristics  and  professional 
competencies  required  for  successful  performance  in  a  specific  work  setting.  The 
results  of  the  job  analysis  help  determine  the  actual  assessment  scales  that  are 
included  in  the  selection  procedures. 

•  Conduct  a  validity  study  to  provide  empirical  support  for  the  accuracy  and  job¬ 
relatedness  of  the  assessment  scales. 

•  Use  results  of  the  job  analysis  and  validity  study  to  create  a  personality  profile. 

•  Conduct  adverse  impact  analyses  to  ensure  that  cutoff  scores  do  not  discriminate 
against  any  protected  demographic  group  or  class.123 


122  Topiak,  M.E.,  West,  R.F.  &  Stanovich,  K.E.  "The  Cognitive  Reflection  Test  as  a  Predictor  of  Performance  on 
Heuristics-and-Biases  Tasks,"  Memory  &  Cognition,  Vol.  39,  No.  7,  2011,  p.  1275. 

123 Anderson,  M.G.  Personality  Assessment  in  Personnel  Selection,  CPP  Inc.,  Mountain  View,  CA,  2007,  pp.  5-6. 
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Of  the  assessments  reviewed  in  this  report,  only  the  CCTDI  was  promoted  by  its  authors  as 
useful  in  employee  selection  processes.  The  CCTDI  user’s  manual  states: 

The  attitudes  and  attributes  measured  by  the  CCTDI  have  been  endorsed  by  employers 
around  the  world  as  the  attributes  desired  in  representatives  of  their  companies  and 
agencies.  Some  agencies  have  explicit  objectives  aimed  at  building  a  critical  thinking 
culture,  and  use  scores  on  the  CCTDI  in  conjunction  with  staff  development  workshops 
to  engender  group  discussions  about  the  value  of  critical  thinking.124 

The  Canadian  Intelligence  Community  (IC)  is  investigating  the  value  of  two  assessments 
mentioned  in  this  report:  the  NFC  Scale  and  the  Need  for  Closure  Scale  (NFCC),  referred  to  as 
the  NFCC  in  the  following  quote: 

Given  the  nature  of  these  individual  differences  and  the  distinct  requirements  of 
intelligence  analysis,  we  would  expect  more  successful  analysts  to  score  higher  than 
average  in  the  NFC  scale  and  lower  in  the  NFCC  scale.  McLellan  and  Mandel  (2010b) 
have  begun  to  assess  Canadian  intelligence  analyst  trainees  on  these  two  individual 
differences  measures,  along  with  a  measures  of  accuracy  and  coherence  based  on  the 
Decision  Making  Competence  index  (de  Bruin  et  al.  2007).  The  aim  of  their  field 
research  is  to  test  whether  or  not  those  individual  differences  predict  coherence  and 
accuracy,  and  to  establish  baseline  levels  for  the  community,  as  compared  to  the  general 
population.  To  the  best  of  our  knowledge,  there  have  been  no  efforts  to  date  to  correlate 
these  individual  differences  with  perfonnance  measures  with  a  sample  of  intelligence 
personnel.  If  proven  to  be  reliable  predictors  of  analysts’  decision  making  performance, 
then  the  NFC  and  NFCC  scales  may  be  usefully  employed  in  the  analyst  selection 
process.  The  NFC  and  NFCC  scales  may  complement  the  TSD-PI  measure  recommended 
by  Girard  (2010)  for  the  selection  process  of  Intelligence  Officers  and  Intelligence 

125 

Operators  in  the  Canadian  Forces. 

An  initial  report  of  results  was  presented  at  the  Annual  Southern  Ontario  Behavioral  Decision 
Research  Conference,  Toronto,  Ontario,  Canada  in  2010.  Note  that  Decision  Making 
Competence  (DMC)  refers  to  an  assessment  which  was  used  to  provide  the  outcome  benchmark 
of  cognitive  performance. 

Conclusion: 

•  Using  a  sample  of  intelligence  trainees,  the  current  study  revealed  the  predictive 
value  of  individual  differences  in  cognitive  styles  on  a  measure  of  decision  making 
that  is  highly  relevant  to  the  tradecraft  of  intelligence  analysis. 

•  NFC  and  NFCC — particularly  the  Closemindedness  -  predicted  performance  on 
framing  items  of  the  DMC. 

•  These  findings  extend  prior  research  demonstrating  the  association  between 
cognitive  style  on  judgment  and  decision  making  (e.g.,  Mandel,  2005;  Stanovich  & 
West,  2000). 


124  Facione,  P.A.  &  Facione,  N.C.  California  Critical  Thinking  Disposition  Inventory  (CCTDI)  Test  Manual, 
California  Academic  Press,  Millbrae,  CA,  2010,  p.  6. 

125  Derbentseva,  N.,  McLellan,  L.  &  Mandel,  D.R.  Issues  in  Intelligence  Production:  Summary  of  Interviews  with 
Canadian  Managers  of  Intelligence  Analysts,  TR  2010-144,  Defence  R&D  Canada,  Toronto,  ON,  2010,  pp.  47-48. 
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•  Findings  from  this  study  may  help  to  inform  practices  for  selecting  intelligence 
analysts  within  the  Canadian  Forces,  and  the  Canadian  intelligence  community 
more  broadly.126 


The  US  IC  is  calling  for  a  more  deliberate  approach  to  analyst  selection.  Although  not 
specifically  mentioning  thinking  dispositions,  Kozlowski  wrote  about  individual  differences  in 
the  areas  of  knowledge,  skills,  abilities,  and  other  characteristics  (KSAO).  He  appears  to  be 
confirming  the  need  for  the  steps  outlined  by  CPP,  Inc. 

People  differ  from  one  another  on  a  wide  range  of  characteristics.  Individuals  differ  on 
demographic  features  (e.g.,  age,  sex,  race),  abilities  (e.g.,  cognitive,  physical),  and 
preferences  (e.g.,  personality,  values).  The  focus  from  a  human  resources  perspective  is 
on  differences  in  KSAOs  (e.g.,  personality,  interests,  and  values)  that  are  linked  to 
differences  in,  for  example,  educational  attainment,  vocational  preferences,  job 
performance,  and  career  success.  At  the  most  basic  level,  KSAOs  are  individual 
differences  that  contribute  to  job  performance.127 

Kozlowski  further  explained  KSAOs: 

KSAOs  can  be  divided  into  those  that  are  stable  and  those  that  are  malleable.  Stable 
KSAOs  include  factors  such  as  cognitive  ability,  personality,  and  values  that  are 
relatively  enduring  across  the  span  of  adult  development.  Malleable  KSAOs  include 
factors  such  as  domain  knowledge,  job-specific  skills,  and  motivational  characteristics. 
For  example,  cognitive  ability,  which  is  a  generalized  predictor  of  learning  and 
performance  effectiveness  and  has  a  high  genetic  component,  is  very  stable  across  a 

person’s  career  (Lyons  et  ah,  2009),  whereas  domain  knowledge  and  job-specific  skills 
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accrue  over  time  through  experience  and  training. 

He  emphasized  the  importance  of  understanding  the  KSAOs  that  are  needed  for  the  organization 
through  a  job  analysis.  This  is  the  first  step  of  the  CPP  process: 

Scientific  selection  is  a  well-developed  and  proven  methodology  and  set  of  practices  that 
have  been  in  general,  though  by  no  means  universal,  use  for  a  century.  The  essence  of 
selection  is  to  assess  applicant  KSAOs  that  are  predictive  of  future  job  performance  and 
then  to  hire  the  best  applicants.  The  development  of  a  selection  system  has  several  key 
steps.  Job  analysis  is  the  bedrock  of  selection  system  development.  It  is  a  systematic 
process  to  identify  the  important  and  critical  task  behaviors  that  comprise  a  job  and  the 
underlying  KSAOs  required  for  effective  job  performance.129 


126  McLellan,  L.  &  Mandel,  D.R.  "Cognitive  Styles  Predict  Accuracy  and  Coherence  in  Intelligence  Analyst's 
Judgments  [Poster  Presentation],"  Sixth  Annual  Southern  Ontario  Behavioral  Decision  Research  Conference, 
Toronto,  ON,  May  7,  2010,  p.  2. 

127  Fingar,  T.  "Analysis  in  the  U.S.  Intelligence  Community:  Missions,  Masters,  and  Methods,"  in  Intelligence 
Analysis:  Behavioral  and  Social  Scientific  Foundations,  eds.  B.  Fischhoff  &  C.  Chauvin,  National  Academies  Press, 
Washington,  D.C.,  2011,  p.  277. 

128  Fingar,  p.  277. 

129  Fingar,  p.  284. 
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4.3.11.  Summary 

Thinking  dispositions  can  be  assessed  and  a  variety  of  different  assessments  exist  that 
organizations  can  use  for  employee  selection.  Although  not  all  of  the  assessments  were  proven 
to  have  outstanding  reliability  and  validity,  many  were  proven  to  be  sufficiently  reliable  and 
valid  that  they  have  potential  for  employee  selection  purposes. 

The  many  thinking  disposition  assessments  provide  a  useful  starting  point  for  the  intelligence 
community  to  identify  the  specific  assessment  or  combination  of  assessments  that  could  predict 
potential  analytic  success.  The  studies  reported  in  this  section  and  in  the  previous  section 
(demonstrating  the  impact  of  thinking  dispositions  on  cognitive  performance)  provide  models  for 
studies  designed  to  identify: 

•  What  specific  thinking  dispositions  provide  the  most  useful  prediction  of  analytic 
success? 

•  Which  of  these  assessments  are  most  useful  in  identifying  individuals  who  have  the 
cognitive  dispositions  that  lead  to  contribute  to  effective  analytic  performance? 

In  the  next  section  of  the  report  we  will  address  a  part  of  the  first  step  of  the  CPP,  Inc.  process 
and  the  process  described  by  Kozlowski.  We’ll  investigate  the  traits  and  characteristics  of 
successful  analysts  to  see  how  these  map  to  thinking  dispositions. 

4.4  Thinking  Dispositions  Compared  to  the  Traits  and  Characteristics  of  Effective 
Intelligence  Analysts 

With  our  discussion  of  thinking  disposition  concepts  and  value  now  complete,  we  turn  our 
attention  to  how  thinking  dispositions  might  relate  to  intelligence  analysis.  What  are  the 
accepted  traits  and  characteristics  of  intelligence  analysts  and  how  do  these  compare  to  thinking 
dispositions? 

Although  authors  and  organization  have  compiled  many  lists  of  analyst  characteristics  or 
attributes  over  the  years,  only  one  document  in  the  intelligence  analysis  literature  explicitly 
discusses  thinking  dispositions.  David  Moore,  in  Critical  Thinking  and  Intelligence  Analysis, 
describes  dispositions  related  to  critical  thinking.  Moore  quoted  from  the  list  of  dispositions 
provided  by  the  Foundation  for  Critical  Thinking  and  by  Facione  and  noted: 

Both  sets  closely  match  the  characteristics  of  successful  intelligence  analysts  identified 
by  Lisa  Krizan  and  the  author  in  their  work  on  intelligence  analysts'  core  competencies. 
Krizan  and  the  author  observe  that  successful  intelligence  analysts  are  insatiably  curious. 
Fascinated  by  puzzles,  their  high  levels  of  self-motivation  lead  them  to  observe  and  read 

131 

voraciously,  and  to  take  fair-minded  and  varied  perspectives. 

In  this  section  of  the  report,  we  explore  how  the  IC  describes  analyst  characteristics.  We  then 
compare  these  descriptions  with  what  we  have  learned  about  thinking  dispositions. 


130  See  Table  1  of  this  report  for  the  thinking  disposition  lists  referenced  in  this  paragraph. 

131  Moore,  D.T.  Critical  Thinking  and  Intelligence  Analysis,  Joint  Military  Intelligence  College,  Washington,  D.C., 

2006,  p.  16. 
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4.4.1.  Intelligence  Analyst  Characteristics 

Many  documents  described  the  skills  needed  and  tasks  perfonned  by  intelligence  analysts.  I 
concentrated  on  finding  descriptions  of  cognitive  characteristics  that  appeared  to  be  similar  to 
thinking  dispositions  or  that  were  focused  specifically  on  cognitive  aspects  of  analysis. 

Clauser  and  Weir  described  characteristics  of  the  intelligence  researcher  in  1976.  These 
characteristics  resemble  our  lists  of  thinking  dispositions  and  include: 

•  Reasoning  ability 

•  Accuracy 

•  Intellectual  honesty 

•  Open-mindedness 

•  Skepticism 

•  Detachment 

•  Patience,  diligence,  perseverance 

•  •  132 

•  Imagination 

A  1976  IC  research  project  identified  the  following  characteristics  of  the  ideal  intelligence 
analyst.  The  purpose  of  the  study  was  to  develop  an  understanding  of  the  cognitive  processes 
that  are  required  for  intelligence  analysis.  The  attributes  in  this  study  have  little  in  common  with 
thinking  dispositions. 

The  attributes  of  the  ideal  analyst  offered  by  analysts  interviewed  during  the  course  of 
this  project  are  summarized  as  follows.  The  ideal  analyst: 

•  Is  a  technologist 

•  Is  focused  (either  a  specialist  or  generalist,  not  both). 

•  Is  an  information  entrepreneur  (as  described  earlier). 

•  Is  comfortable  with  changing  roles  as  apprentice,  peer,  trainer,  or  consultant. 

•  Can  communicate  (written  and  oral). 

•  Is  a  detective 

•  Is  imaginative 

•  Is  self-starting,  self-organizing 

•  Has  a  profession  (Intelligence  analysis). 

•  Has  related  hobbies  or  technology  interests. 

•  Prefers  analysis  to  management. 

•  Can  perfonn  multiple,  concurrent  activities. 

•  Is  self-confident. 

133 

•  Has  a  photographic  memory. 

132  Clauser,  J.K.  &  Weir,  S.M.  Intelligence  Research  Methodology’:  An  Introduction  to  Techniques  and  Procedures 
for  Conducting  Research  in  Defense  Intelligence,  HRB-Singer,  State  College,  PA,  1976,  pp.  48-60. 

133  Sullivan,  M.  "Internal  Recruiting  for  an  Intelligence  Analyst,"  SCIP. online,  Vol.  1,  No.  13,  2002,  pp.  5-5  -  5-6. 
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Fischl  and  Gilbert  investigated  whether  the  use  of  cognitive  assessments  might  help  in  selecting 
intelligence  analysts.  They  identified  the  following  personal  attributes  that  might  predict 
analytic  success: 

•  High  level  reasoning  ability 

•  Inductive  reasoning 

•  Intellectual  flexibility 

•  Writing  skill 

•  Memory 

•  Intellectual  curiosity 

•  Deliberateness,  carefulness 

•  Interpersonal  skill 

•  Achievement  motivation 

•  Self-discipline 

•  Perseverance134 

Lisa  Krizan  included  a  portrait  of  intelligence  analysts  in  Intelligence  Essentials  for  Everyone. 
She  used  the  cognitive  styles  assessment  called  the  Myers-Briggs  Type  Indicator  (MBTI)to 
describe  characteristics  of  analysts: 

The  third  component  of  the  intelligence  analyst  profile,  personality  traits,  addresses  the 
individual’s  preferences  for  behaving  in  certain  ways  under  specific  conditions.  Adults 
tend  to  exhibit  the  same  set  of  behavior  preferences  consistently  in  familiar  situations. 
This  behavior  pattern  may  be  identified  as  a  personality  type.  One  well-known 
instrument  for  identifying  an  individual’s  personality  type  is  the  Myers-Briggs  Type 
Indicator  (MBTI).104  The  following  discussion  of  the  intelligence  analyst’s  personality  is 

ITS 

based  upon  MBTI  research. 

A  significant  percentage  (2 1  percent)  of  those  who  choose  to  pursue  employment  in 

national  security  intelligence  tend  to  express  the  following  behavior  preferences: 

orientation  to  the  inner  world  of  ideas  rather  than  the  outer  world  of  things  and  people, 

tendency  to  gather  factual  information  through  the  senses  rather  than  inspiration, 

proclivity  to  make  decisions  on  the  basis  of  logic  rather  than  emotion,  and  an  eagerness  to 

1 

seek  closure  proactively  instead  of  leaving  possibilities  open. 

Moore  and  Krizan  wrote  a  series  of  articles  describing  analyst  characteristics.  The  first  was 
written  in  2001, 137  the  last  in  2005. 138  All  of  their  articles  included  the  same  model  of 
competencies,  as  shown  in  Figure  1. 


134  Fischl,  M.A.  &  Gilbert,  A.C.F.  Selection  of  Intelligence  Analysts,  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  Alexandria,  VA,  1983,  p.  386. 

135  Krizan,  L.  Intelligence  Essentials  for  Even’ one,  Joint  Military  Intelligence  College,  Washington,  D.C.,  1999,  p. 
58. 

136  Krizan,  p.  59. 

137  Moore,  D.T.  &  Krizan,  L.  "Intelligence  Analysis:  Does  NSA  Have  What  it  Takes?"  Cryptologic  Quarterly,  Vol. 
20,  No.  1-2,  2001,  pp.  1-32. 

138  Moore,  D.T.,  Krizan,  L.  &  Moore,  E.J.  "Evaluating  Intelligence:  A  Competency-Based  Model,"  International 
Journal  of  Intelligence  and  Counterintelligence,  Vol.  18,  No.  2,  2005,  pp.  204-220. 
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Figure  1:  Core  Competencies  of  Intelligence  Analysts 

James  Simon  emphasized  the  importance  of  tolerance  of  ambiguity  in  “Analysis,  Analysts,  and 
Their  Role  in  Government  and  Intelligence:” 

I  would  argue  that  the  greatest  predictor  of  your  success  as  an  analyst  (not  success;  that’s 
a  poor  tenn,  because  success  is  something  you  have  to  define  for  yourself)  has  to  do  with 
how  you  deal  with  ambiguity.140 

Simon  described  why  he  thought  this  thinking  disposition  was  so  important  in  analysis: 

What  we’re  looking  for  are  people  who  care  a  great  deal  about  facts,  but  are  willing  to 
reach  a  conclusion  without  all  the  facts.  If  you  wait  to  have  all  the  facts  before  you  give 
us  an  answer,  then  we  will  wait  for  you  forever.  There  are  never  all  the  facts  in  the  world 
of  analysis.  At  some  point,  you  simply  must  write  something  or  tell  someone  something, 
and  that  brings  in  judgment,  accuracy,  and  all  the  other  things  I’ve  been  talking  about. 

Y ou  have  to  be  tolerant  enough  of  ambiguity  to  be  willing  to  speak  or  write  without 
possession  of  all  the  facts.141 

Although  open-mindedness  appeared  on  many  lists  of  analyst  characteristics,  O’Connor  provided 
a  contrary  view  of  the  disposition  to  be  open  minded: 

Intelligence  analysis  doesn’t  involve  an  open  mind.  There’s  no  such  thing  as  an  open 
mind,  no  matter  how  you  define  it.  Preconceptions  are  inevitable.  Intelligence  analysts 
obtain  objectivity  by  making  basic  assumptions  and  reasoning  as  explicitly  as  possible. 


y  Moore  &  Krizan,  p.  206. 

140  Simon,  J.M.  Analysis,  Analysts,  and  their  Role  in  Government  and  Intelligence,  Center  for  Information  Policy 
Research,  Harvard  University,  Cambridge,  MA,  2003,  p.  7. 

141  Simon,  p.  8. 
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Validity  is  obtained  by  the  two-fold  process  of  self-examination  (regarding  the 
assumptions  made)  and  making  your  work  challengeable  by  other  analysts  (reasoning 
explicitly).142 

Derbentseva  et  al.  interviewed  seven  Canadian  intelligence  professionals  to  develop  a  list  of 
skills  and  characteristics  needed  for  successful  analysts.  Their  list  included: 

•  Ability  to  deal  with  uncertainty 

•  Accurate  memory 

•  Flexibility 

•  Integrity  and  moral  courage 

•  Self-starter 

•  Thirst  for  knowledge 

•  Thoroughness143 

The  authors  also  noted  the  potential  value  of  recognizing  the  presence  or  absence  of  the  need  for 
cognition  and  the  need  for  cognitive  closure  during  the  analyst  selection  process. 

It  is  interesting  to  compare  intelligence  analyst  characteristics  from  a  business  perspective  with 
those  identified  by  the  IC.  Mark  Sullivan,  Manager  of  Strategic  Analysis  with  General  Motors, 
provided  a  list  of  characteristics  for  intelligence  analysts  in  the  business  community: 

But,  a  good  intelligence  analyst  is  more  than  just  an  analyst.  He  or  she  should  also  have  a 
set  of  fairly  unique  personal  attributes.  Among  these  are: 

•  The  ability  to  deal  with  a  high  level  of  ambiguity  and  lack  of  structure. 

•  The  ability  to  be  a  self-starter. 

•  The  ability  to  be  comfortable  and  articulate  with  people  in  the  organization  who  are 
considerably  more  senior  then  he  or  she  is. 

•  The  ability  to  get  the  customer  to  define  what  is  really  needed  and  why. 

•  The  humility  to  understand  that,  as  a  support  person,  an  intelligence  analyst  is  not 
an  advocate. 

•  The  personal  fortitude  to  push  an  intelligence  finding  so  that  it  gets  the  attention  it 
deserves  by  senior  managers  —  even  if  this  means  crossing  the  formal  hierarchy  of 
the  company.144 

4.4.2.  Comparing  Analyst  Characteristics  with  Thinking  Dispositions 

Table  3  provides  a  comparison  of  intelligence  analyst  characteristics  with  thinking  dispositions. 
Considerable  commonality  exists  between  the  two  categories. 


142  O'Connor,  T.  Intelligence  Analysis.  Lecture  Notes.  2004,  p.  3. 

143  Derbentseva,  N.,  McLellan,  L.  &  Mandel,  D.R.  Issues  in  Intelligence  Production:  Summary’  of  Interviews  with 
Canadian  Managers  of  Intelligence  Analysts,  TR  2010-144,  Defence  R&D  Canada,  Toronto,  ON,  2010,  pp.  43-44. 

144  Sullivan,  M.  "Internal  Recruiting  for  an  Intelligence  Analyst,"  SCIP. online,  Vol.  1,  No.  13,  2002,  p.  2. 

53 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 

88ABW-20 14-2455;  Cleared  21  May  2014 


Table  3:  Comparison  of  Analyst  Characteristics  and  Thinking  Dispositions 


Source 

Analyst  Characteristic 

Related  Thinking  Disposition 

Clauser  &  Weir 

Intellectual  honesty 
Open-mindedness 

Skepticism 

Patience,  diligence,  perseverance 

Intellectual  honesty 

Open-mindedness 

Intellectual  skepticism,  focus  on 
evidence 

Persistent 

Fischl  &  Gilbert 

Intellectual  flexibility 

Intellectual  curiosity 
Deliberateness,  carefulness 
Self-discipline 

Perseverance 

Flexibility,  flexible  thinking 

Intellectual  curiosity 

Systematic,  intellectually  careful 
Intellectual  discipline 

Persistent 

Moore  &  Krizan 

Insatiably  curious 

Self  Motivated 

Fascinated  by  puzzles 

Observes  voraciously 

Reads  voraciously 

Fruitfully  obsessed 

Takes  variable  perspectives 
Concentrates  intensely 

Questions  conventions 

Intellectual  curiosity 

Intellectual  discipline 

Need  for  cognition 

Intellectually  curious 

Intellectually  curious 

Truthseeking 

Intellectual  empathy,  fairmindedness 
Think  extensively  about  a  problem 
before  responding,  Intellectually  careful 
Intellectual  autonomy,  truth-seeking 

Simon 

Tolerance  of  ambiguity 

Tolerance  of  ambiguity 

Derbentseva 

Ability  to  deal  with  uncertainty 
Flexibility 

Integrity  and  moral  courage 

Self-starter 

Thirst  for  knowledge 
Thoroughness 

Need  for  cognition 

Need  for  cognitive  closure 

Tolerance  of  ambiguity 

Flexibility,  flexible  thinking 

Intellectual  integrity,  truth-seeking, 
Intellectual  honesty 

Intellectual  discipline 

Intellectual  curiosity 

Systematic,  intellectually  careful 

Need  for  cognition 

Need  for  cognitive  closure 

Sullivan 

The  ability  to  deal  with  a  high 
level  of  ambiguity  and  lack  of 
structure. 

The  ability  to  be  a  self-starter. 

Tolerance  of  ambiguity 

Intellectual  discipline 

4.4.3.  Summary 

Our  list  of  thinking  dispositions  does  not  address  all  of  the  competencies  needed  for  analysis. 
Analysts  also  require  specific  skills,  abilities,  and  knowledge.  However,  where  scholars 
described  disposition-like  attributes  of  intelligence  analysts,  significant  overlap  exists  between 
intelligence  analyst  attributes  and  thinking  dispositions.  It  is  clear  that  the  thinking  dispositions 
we  have  identified  have  been  recognized  as  important  contributors  to  analytic  success. 


54 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-20 14-2455;  Cleared  21  May  2014 


Now  that  we’ve  explored  how  thinking  dispositions  can  be  potentially  valuable  contributors  to 
effective  analysis,  we  investigate  whether  we  can  strengthen  the  thinking  dispositions  of 
analysts. 

4.5  Developing  and  Improving  Thinking  Dispositions 

Recognizing  how  important  strong,  positive  thinking  dispositions  are  to  effective  thinking,  we 
now  investigate  the  question  of  how  to  strengthen  the  disposition  to  think  effectively.  We 
address  this  objective: 

•  How  is  an  individual’s  thinking  disposition  initially  established  or  subsequently 
changed? 

o  What  factors  influence  the  development  of  various  thinking  dispositions? 
o  Can  thinking  dispositions  be  improved  with  training  and  education  or  other 
factors  such  as  life  experience  or  maturity?  What  evidence  exists  for  this 
improvement? 

Many  scholars  identified  the  culture  in  which  an  individual  lives  as  an  important  contributor  to 
developing  thinking  dispositions.  For  example,  a  family  or  society  that  discourages  questioning 
and  investigation  causes  the  individual  living  in  that  environment  to  avoid  those  behaviors. 
Perkins,  Jay,  and  Tishman  described  the  impact  of  the  cultural  environment  this  way: 

Emphasis  on  the  dispositional  side  of  thinking  implies  that  development  involves  more 
than  just  the  growth  of  cognitive  abilities.  The  idiom  of  traditional  developmental 
psychology,  with  its  focus  on  the  individual’s  cognitive  capacities,  concentrates  on 
internally-propelled  change.  The  growth  of  dispositions,  however,  cannot  be  explained 
by  looking  at  the  individual  as  a  self-contained  system.  Because  dispositions  are 
grounded  in  belief  systems,  values,  and  attitudes  as  much  as  in  cognitive  structures,  we 
need  a  culturally  based  account  of  their  development.  An  alternative  tradition  in 
developmental  psychology,  emanating  from  the  work  of  Vygotsky,  acknowledges  the 
role  of  social  and  cultural  influence  on  development.  Vygotsky  (1978)  argues  that 
thinking  is  a  social  activity,  initially  shared  between  people  but  gradually  internalized  in 
the  individual.  Cultural  artifacts  (language  in  particular)  are  experienced  through 
interaction,  re-enacted  by  the  individual,  and  eventually  internalized.  In  a  similar  vein, 
Brown,  Collins,  and  Duguid  (1989)  call  learning  a  matter  of  enculturation:  by  observing 
and  living  within  a  particular  culture  people  gradually  start  to  adopt  the  behavior  and 
belief  systems  of  the  culture.145 

They  go  on  to  say: 

At  the  interpersonal  level,  children  may  develop  thinking  dispositions  through  social 
interaction  with  more  experienced  members  of  society  (Vygotsky,  1978).  For  example,  a 
child  whose  family  members  model  acceptance  of  different  viewpoints  may  encourage 
the  development  of  the  disposition  to  be  broad  and  adventurous.146 

Although  thinking  dispositions  are  attitudinal,  the  majority  of  scholars  who  wrote  about  thinking 
dispositions  described  them  as  something  that  could  be  taught  and  developed,  similar  to  teaching 


145  Perkins,  D.N.,  Jay,  E.  &  Tishman,  S.  "Beyond  Abilities:  A  Dispositional  Theory  of  Thinking,"  The  Merrill- 
P aimer  Quarterly,  Vol.  39,  No.  1,  1993,  pp.  16-17. 

146  Perkins  et  al.,  p.  17. 
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and  developing  skills.  However,  because  thinking  dispositions  refer  to  inclinations  that  are 
habitual  and  ingrained,  a  simple  introduction  to  the  concepts  of  thinking  dispositions  is  unlikely 
to  result  in  the  needed  attitudinal  or  behavior  change. 

I  found  two  approaches  to  teaching  thinking  dispositions: 

•  Creating  a  culture  in  the  classroom  and  in  work  or  life  that  supports  and  strengthens 
positive  thinking  dispositions  in  general.  The  cultural  approach  emphasizes  two 
aspects:  establishing  an  expectation  that  positive  thinking  dispositions  will  be 
demonstrated  and  enabling  ongoing  opportunities  to  reinforce  the  dispositions  and 
make  them  habitual. 

•  Explicitly  teaching  thinking  dispositions  -  developing  a  curriculum  focused 
specifically  on  strengthening  and  practicing  one  or  more  dispositional  attitudes  that 
support  effective  thinking. 

4.5.1.  Creating  a  Thinking  Culture 

The  importance  of  the  thinking  culture  was  recognized  as  early  as  1901.  Nathan  Schaeffer 
described  the  importance  of  the  teacher  in  his  book  Thinking  and  Learning  to  Think : 

Right  here  the  teacher  who  is  an  artist  finds  the  opportunity  for  the  display  of  his  highest 
skill.  It  is  his  privilege  to  direct  the  flights  and  the  perchings  of  the  youthful  mind.  He 
can  shape  the  thoughts  and  their  sequence.  He  can  cause  the  intellect  to  move  from  the 
reason  to  its  consequence,  or  in  the  reverse  direction  if  that  be  more  natural  or  more 
appropriate.  He  can  guide  the  thought  from  cause  to  effect,  from  the  whole  to  the  parts, 
from  the  general  to  the  particular,  from  the  end  to  the  means,  from  the  design  to  its 
execution;  or  a  movement  the  other  way  is  possible  in  each  of  these  categories.  While 
thus  choosing  the  direction  which  thought  shall  take,  he  can  select  the  objects  upon  which 
it  shall  tarry.  This  directing  influence  he  will  often  exert  when  he  is  not  aware  of  it.  His 
own  habits  of  mind  will  be  reflected  in  the  mental  life  of  his  pupils.  There  was  profound 
philosophy  in  the  reply  of  a  gifted  author  who,  when  asked  by  his  daughter  what  she 
should  study,  said,  "I  am  more  concerned  about  the  teachers  under  whom  you  study  than 
about  the  branches  of  study  which  you  may  select."  Habits  of  thought  depend  far  more 
upon  the  teacher  than  upon  the  text-book,  upon  the  quality  of  the  instruction  than  upon  its 

i  147 

general  content. 

John  Dewey  also  emphasized  the  importance  of  the  teacher  in  creating  a  culture  for  thinking: 

The  Training  of  Thought  Is  Indirect 

For  these  reasons,  the  problem  of  method  in  forming  habits  of  reflective  thought  is  the 
problem  of  establishing  conditions  that  will  arouse  and  guide  curiosity;  of  setting  up  the 
connections  in  things  experienced  that  will  on  later  occasions  promote  the  flow  of 
suggestions,  create  problems  and  purposes  that  will  favor  consecutiveness  in  the 
succession  of  ideas.  These  topics  will  be  considered  more  at  length  later,  but  an 
illustration  or  two  drawn  from  failure  to  secure  proper  conditions  will  indicate  more 
clearly  what  is  meant.  Children  are  hushed  up  when  they  ask  questions;  their  exploring 
and  investigating  activities  are  inconvenient  and  hence  they  are  treated  like  nuisances; 
pupils  are  taught  to  memorize  things  so  that  merely  one-track  verbal  associations  are  set 


147  Schaeffer,  N.C.  Thinking  and  Learning  to  Think,  L.  B.  Lippincott  Co.,  Philadelphia  PA,  1901,  p.  219. 
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up  instead  of  varied  and  flexible  connections  with  things  themselves;  no  plans  and 
projects  are  provided  that  compel  the  student  to  look  ahead  and  foresee  and  in  the 
execution  of  which  the  accomplishment  of  one  thing  sets  up  new  questions  and  suggests 
new  undertakings.  The  teacher  may  devise  special  exercises  intended  to  train  thinking 
directly,  but  when  these  wrong  conditions  exist,  special  exercises  are  doomed  to  be  futile. 
The  training  of  thought  can  be  attained  only  by  regulating  the  causes  that  evoke  and 
guide  it.148 

Peter  Facione  emphasized  the  importance  of  culture: 

Conversations  with  many  around  the  world  seeking  to  develop  or  evaluate  programs  to 
nurture  positive  habits  of  mind  lead  us  to  urge  that  modeling  and  cultivating  a  healthy 
culture  of  learning  and  inquiry  may  be  as  important,  or  more  important,  than  any 
instructional  content.149 

Barry  Beyer  provided  a  four-part  model  for  creating  a  classroom  culture  that  strengthens  positive 
thinking  dispositions: 

Teachers  can  use  at  least  four  techniques  to  foster  development  of  attitudes  associated 
with  thinking: 

1.  Model  behaviors  that  demonstrate  the  desired  dispositions. 

Teachers,  in  their  teaching,  should  suspend  judgments  until  they  have  as  much  relevant 
information  as  possible  and  should  make  deliberate,  explicit  efforts  to  secure  such 
information.  They  should  deliberately  seek  out  a  variety  of  points  of  view  on  an  issue  or 
topic.  They  should  articulate  a  number  of  alternatives  in  decision-making  situations 
before  considering  any  and  making  choices.  They  should  give  their  reasoning  for  their 
assertions  and  decisions.  In  so  doing,  teachers  should  explain  to  their  students  why  they 
are  doing  these  things,  and  they  need  to  be  sure  that  students  recognize  this 
demonstrating  of  important  aspects  of  thinking. 

2.  Insist  on  student  behavior  that  reflects  the  dispositions  sought. 

Teachers  should  require  students  to  exhibit  behaviors  demonstrate  thinking,  just  as  they 
do.  Students  should  be  required  to  give  reasons  for  their  claims  and  decisions  and  to  ask 
others  to  do  the  same,  to  generate  alternatives  before  making  choices,  to  seek  out  and 
explore  a  variety  of  points  of  view,  to  withhold  judgment,  and  so  on.  Practice  of  such 
behaviors,  even  if  at  the  insistence  of  a  teacher,  can,  in  time,  lead  to  internalization  of  the 
values  implicit  in  these  behaviors. 

3.  Engage  students  in  repeated  activities  that  require  use  of  these  disposition-based 
behaviors. 

Learning  activities  must  consistently  and  continuously  require  students  to  exhibit 
behaviors  related  to  effective  thinking.  One  or  two  cases,  instances,  or  opportunities  to 
do  so  over  the  course  of  a  semester  or  year  are  not  sufficient  even  to  begin  to  develop 
such  dispositions.  Teachers  must  design  and  carry  out  activities  in  which  students 
repeatedly  seek  out  and  discuss  a  variety  of  points  of  view,  collect  additional  data, 


148  Dewey,  J.  How  We  Think:  A  Restatement  of  the  Relation  of  Reflective  Thinking  to  the  Educative  Process,  2nd  ed., 
Regnery,  Chicago,  IL,  1933,  pp.  56-57. 

149  Facione,  P.A.,  Facione,  N.C.  &  Giancarlo,  C.A.F.  Professional  Judgment  and  the  Disposition  Toward  Critical 
Thinking,  California  Academic  Press,  Millbrae,  CA,  1997,  p.  7. 
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suspend  judgment,  and  choose  from  a  number  of  alternatives.  These  and  related 
behaviors  must  be  practiced  in  appropriate  places  over  and  over  again  across  many  grade 
levels  and  in  many  subjects. 

4.  Reinforce  behaviors  that  demonstrate  the  appropriate  dispositions. 

Behaviors  illustrative  of  effective  thinking  can  be  reinforced  by  explaining  and 
demonstrating  their  value  as  well  as  by  offering  praise,  grades,  and  other  reinforcements 
for  them.  Teachers  need  to  take  care  in  so  doing,  however,  to  ensure  that  their 
reinforcements  clearly  relate  to  the  behavior,  not  to  the  students(s)  exhibiting  it.  Few,  if 
any,  students  can  be  or  even  may  wish  to  be  like  another  individual,  but  all  can  exhibit  a 
valued  behavior.150 

Tishman,  Jay,  and  Perkins  made  an  explicit  call  for  creating  a  culture  that  nurtures  and 
strengthens  positive  thinking  dispositions  in  “Teaching  Thinking  Dispositions:  From 
Transmission  to  Enculturation.”  The  authors  describe  traditional  teaching  methods: 

Much  of  conventional  instruction  reflects  a  tacit  conception  of  the  teaching-learning 
process  that  might  be  termed  the  transmission  model.  The  essence  of  this  model  is  easily 
stated:  The  teacher's  role  is  to  prepare  and  transmit  information  to  learners.  The  learners' 
role  is  to  receive,  store,  and  act  on  this  information.  15 

The  authors  note  the  impact  this  traditional  approach  has  on  thinking  dispositions: 

One  reason  to  turn  to  an  enculturation  model  is  that  some  sort  of  culture  in  the  classroom 
always  exists:  We  are  enculturating  whether  we  recognize  it  or  not,  so  we  may  as  well 
take  heed  and  enculture  what  we  want.  For  example,  inevitably,  the  transmission  model 
enculturates  certain  sorts  of  inclinations  and  sensitivities.  Consider  a  rather  rigid  version 
of  the  model  where  students'  role  is  to  sit  quietly  and  receive  the  information  they  need 
for  the  test.  In  such  an  environment,  an  inclination  to  be  passive  with  respect  to 
knowledge  tends  to  develop.  Students  do  not  become  disposed  to  seek  and  evaluate 
information  on  their  own;  rather,  they  learn  to  count  on  the  environment  to  automatically 
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feed  them  information. 

They  propose  a  model  to  explicitly  teach  for  strong,  positive  thinking  dispositions  through 
enculturation: 

In  any  sustained  cultural  context,  be  it  a  third  grade  classroom,  a  family  setting,  or  the 
culture  of  the  workplace,  it  is  useful  to  think  of  enculturation  as  occurring  in  three 
mutually  reinforcing  ways:  through  cultural  exemplars,  cultural  interactions,  and  direct 
instruction  in  cultural  knowledge  and  activities.  These  three  aspects  of  enculturation- 
exemplars,  interaction,  and  instruction-suggest  three  straightforward  guidelines  for 
organizing  teaching:  For  each  thinking  disposition  one  aims  to  enculturate,  one  wants  to 
(a)  provide  exemplars  of  the  disposition;  (b)  encourage  and  orchestrate  student-student 
and  teacher-student  interactions  involving  the  disposition,  and  (c)  directly  teach  the 
disposition.153 


1,0  Beyer,  B.K.  Practical  Strategies  for  the  Teaching  of  Thinking,  Allyn  and  Bacon,  Boston,  MA,  1987,  pp.  211-213. 
151  Tishman,  S.,  Jay,  E.  &  Perkins,  D.N.  "Teaching  Thinking  Dispositions:  From  Transmission  to  Enculturation," 
Theory  into  Practice,  Vol.  32,  No.  3,  1993,  p.  149. 

1,2  Tishman  et  at,  p.  150. 

153  Tishman  et  at,  p.  150. 
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Tishman  et  al.  investigated  the  use  of  explicit  thinking  routines  in  the  classroom  as  tools  for 
building  the  culture  of  thinking  they  recommend.  The  authors  found  that  thinking  routines  could 
instill  the  sensitivity  to  notice  when  thinking  is  needed  and  the  inclination  to  think  when  needed. 
These  thinking  routines  appear  to  provide  a  bridge  between  the  cultural  model  of  teaching 
thinking  dispositions  and  the  explicit  model  of  teaching  thinking  dispositions.  The  authors 
described  how  they  came  to  develop  thinking  routines: 

Consequently,  an  overarching  question  guiding  our  work  has  been:  How  do  we  influence 
and  shape  classroom  culture  to  reflect  a  greater  focus  on  thinking? 

This  principal  question  led  us  to  explore  and  identify  some  of  the  forces  at  work  in 
shaping  culture.  In  his  ethnographic  study  of  thoughtful  classrooms,  Ritchhart  (Ritchhart, 
2002)  identified  eight  forces  that  shape  classroom  culture:  expectations,  time,  modeling, 
routines,  opportunities,  relationships,  physical  environment,  and  language.  From  these 
forces,  we  identified  routines  as  a  potentially  high-leverage  practice  for  creating  more 
thoughtful  classrooms  and  nurturing  students’  thinking  dispositions.  Designing  and 
developing  thinking  routines  for  teachers  to  use  provided  a  very  concrete  intervention 
while  enabling  us  to  touch  on  several  others  of  the  cultural  forces  indirectly.  Thinking 
routines  focus  on  the  establishment  of  structures  that  weave  thinking  into  the  fabric  of  the 
classroom  and  help  to  make  the  thinking  of  everyone  in  the  classroom  more  visible  and 
apparent  (Perkins,  2004),  thus,  they  provide  models  of  thinking.  Routines  also  provide  a 
vehicle  for  incorporating  thinking  language  (Lee,  1997;  Tishman  &  Perkins,  1997)  into 
classrooms  as  well  as  time  and  opportunity  for  students’  thinking.  These  links  to  the 
other  cultural  forces  made  thinking  routines  a  strong  platfonn  from  which  to  influence 
classroom  culture.154 

The  authors  wrote  about  a  number  of  different  thinking  routines;  Figure  2  provides  some 
examples.155 


154  Ritchhart,  R.,  Palmer,  P.,  Church,  M.  &  Tishman,  S.  "Thinking  Routines:  Establishing  Patterns  of  Thinking  in  the 
Classroom,"  American  Educational  Research  Association  Annual  Meeting,  San  Francisco,  CA,  April  7-11,  2006,  pp. 
4-5. 

155  Ritchhart  et  al.,  p.  8. 
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SEE-THINK- WONDER 

CLAIM-SUPPORT-QUESTION 

1 .  What  do  you  see? 

2.  What  do  vou  think  about  that? 

3 .  What  does  it  make  vou  wonder? 

1 .  Make  a  claim  about  the  topic 

2.  Identify  support_for  your  claim 

3.  Ask  a  question  related  to  your  claim 

CONNECT-EXTEND-CHALLENGE 

LOOKING:  10X2 

1 .  How  are  the  ideas  and  information 
connected  to  what  vou  already  know? 

2.  What  new  ideas  did  vou  set  that  extend  vour 
thinking  in  new  directions? 

3.  What  is  still  challenging  or  confusing  for  vou? 
What  questions  or  puzzles  do  you  now  have? 

1 .  Look  image  or  artifact  quietly  for  at  least  30 
seconds.  Let  your  eyes  wander. 

2.  List  1 0  words  or  phrases  about  any  aspect  of  it. 

3.  Repeat  Steps  1  &  2:  Look  at  the  image  or 
artifact  again  and  try  to  list  1 0  more  words  or 
phrases  to  your  list. 

PERCEIVE-KNOW-CARE  ABOUT 

THINK-PUZZLE-EXPLORE 

1 .  What  can  the  nerson  or  thins  nerceive? 

2.  What  might  the  nerson  or  thins  know  about? 

3.  What  might  the  person  or  thing  care  about? 

1 .  What  do  vou  think  vou  know  about  this  tonic? 

2.  What  auestions  or  nuzzles  do  vou  have? 

3.  What  does  the  tonic  make  vou  want  to  explore? 

Figure  2:  Sample  Thinking  Routines 

The  authors  described  the  use  of  these  routines  in  multiple  classrooms  in  several  different 
countries  and  reported  positive  outcomes  from  incorporating  the  routines  into  the  classroom. 

The  routine  referred  to  in  the  next  quote  is  Connect-Extend-Challenge  (CEC).  The  authors  noted 
that  Mark,  the  instructor,  also  had  to  learn  how  to  use  the  routine  effectively  and  how  to  educate 
the  students  explicitly  in  the  use  of  the  routine.  Mark  used  the  routine  to  transform  not  only  the 
culture  of  the  classroom,  but  the  daily  practices  of  student  work. 

Mark’s  observation  of  students’  spontaneous  connection  making  indicates  that  students’ 
disposition  to  think  are  being  enhanced.  As  CEC  becomes  more  a  part  of  their  school 
experience,  students  are  learning  more  than  just  how  to  do  an  assignment  or  answer  the 
teacher’s  questions.  Mark  observed  significant  improvement  in  students’  ability  over 
time,  but  he  also  noticed  that  his  students  were  becoming  more  inclined  to  look  for 
connections,  to  see  connection  making  as  an  important  and  worthwhile  thing  to  do.  In 
addition,  his  comments  above  suggested  that  students  were  spotting  opportunities  for 
connection  making  on  their  own  as  well.  Thus,  over  time  students’  ability,  inclination, 
and  awareness  of  connection  making,  the  disposition  to  make  connections,  was  being 
enhanced  through  Mark’s  use  of  CEC.156 

The  next  example  of  cultural  teaching  comes  from  the  world  of  art.  Nancy  Lampert  noted  that 
education  focused  on  the  visual  arts  includes  specific  activities  that  strengthen  thinking 
dispositions  supportive  of  critical  thinking. 

For  example,  in  studio  critiques  a  key  component  of  an  art  student's  experience  is 
discussion  of  the  strengths,  weaknesses,  successes,  and  failures  of  their  own  work,  as 


156  Ritchhart  et  al.,  p.  29. 
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well  as  the  work  of  fellow  students  and  that  of  artists  outside  the  classroom.  It  is  notable 
that  the  very  root  of  the  word  critique  is  the  same  root  in  the  term  critical  thinking. 

Visual  art  students  think  critically  when  discussing  each  other's  work,  other  artists'  work, 
and  when  solving  the  problems  of  how  to  visually  depict  forms  and  concepts.  No  road 
maps  are  available  to  students  approaching  empty  space  which  must  be  filled  with 
effective  visual  communication,  or  when  interpreting  other  artists'  visual  messages. 

These  processes  include  all  of  the  elements  which  research  has  shown  impact  critical 
thinking:  independent  inquiry,  problem  solving,  interactive  discussion  and  analysis.  Art 
students  continually  think  heuristically  rather  than  algorithmically  when  practicing  their 
discipline  (Amabile,  1996). 157 

Lampert  compared  the  scores  on  the  CCTDI  for  arts  students  and  non-arts  students.  She  found 
that  arts  students  scored  higher  in  three  of  the  scales  of  the  CCTDI:  truth-seeking,  open- 
mindedness,  and  critical  thinking  maturity.  Lampert  found  that  non-arts  majors  scored  higher  on 
the  subscale  of  systematicity. 

The  findings  of  this  study  show  early  indications  that  immersion  in  a  discipline  that 
requires  constant  heuristic  problem  solving,  inquiry,  discussion  and  analysis  may 
condition  the  mind  to  approach  experiences  with  a  disposition  for  accepting  that  there  are 
many  possible  solutions  to  complex  problems  -  in  other  words,  such  a  discipline  may 

158 

condition  the  mind  to  think  critically. 

Attending  college  might  also  be  considered  a  contributor  to  enculturation  about  thinking. 
Pascarella  and  Terenzina,  as  part  of  their  longitudinal  meta- analysis  of  the  impact  of  college, 
identified  the  impact  college  has  on  thinking  dispositions: 

In  addition  to  critical  thinking  skills,  a  new  line  of  inquiry  in  the  1990s  estimated  the 
extent  to  which  students  develop  during  college  in  the  disposition  to  think  critically. 
Across  the  studies  we  uncovered,  the  senior  advantage  over  first  year  students  in  critical 
thinking  disposition  was  also  about  .50  of  a  standard  deviation  (19  percentile  points).159 

Pascarella  and  Terenzina  also  noted  that  the  greatest  gain  came  during  the  first  year  of  college 
rather  than  spread  out  over  all  four  years.  The  final  three  years  supported  and  sustained  positive 
thinking  dispositions  rather  than  further  strengthening  them. 

4.5.2.  Explicitly  Teaching  Thinking  Dispositions 

Several  scholars  wrote  about  experiments  in  explicitly  teaching  thinking  dispositions.  The  first 
experiment  I  found  was  conducted  by  Edward  Glaser  in  1941.  Glaser’s  definition  of  critical 
thinking  emphasized  the  importance  of  the  critical  attitude: 

The  ability  to  think  critically,  as  conceived  in  this  volume,  involves  three  things:  (1)  an 
attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that 


137  Lampert,  N.  "Critical  Thinking  Dispositions  as  an  Outcome  of  Art  Education,"  Studies  in  Art  Education,  Vol.  47, 
No.  3,  2006,  p.  224. 

158  Lampert,  p.  224. 

139  Pascarella,  E.T.  &  Terenzini,  P.T.  How  College  Affects  Students:  A  Third  Decade  of  Research,  Jossey-Bass,  San 
Francisco,  CA,  2005,  p.  205. 
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come  within  the  range  of  one's  experiences,  (2)  knowledge  of  the  methods  of  logical 
inquiry  and  reasoning,  and  (3)  some  skill  in  applying  those  methods.160 

His  program  addressed  all  three  aspects  of  his  definition.  Each  lesson  plan  included  Knowledge 
Objectives,  which  addressed  skills  and  knowledge  as  well  as  Appreciation  Objectives  which 
addressed  thinking  dispositions.  The  following  is  a  sample  of  the  Appreciation  Objectives  from 
Unit  2,  Logic  and  the  Weight  of  Evidence: 

B.  Appreciation  Objectives 

1 .  The  recognition  of  the  general  need  of  evidence  for  what  we  or  others  believe  or 
question;  appreciation  of  the  need  for  having  ground  for  our  opinions,  and  of  asking  for 
evidence  before  believing  assertions  from  others  on  matters  which  are  not  well 
established. 

2.  Appreciation  of  the  fact  that  in  daily  life  we  must  frequently  act  and  make  judgments 
about  matters  where  the  evidence  one  way  or  another  is  far  from  conclusive,  and  that 
while  we  must  often  act  quickly  and  decisively,  we  must  hold  our  minds  open  to  other 
evidence  which  may  later  turn  up. 

3.  Respect  for  reason  and  truth  as  determined  by  the  weight  of  evidence.161 

Glaser  used  six  different  methods  to  evaluate  the  impact  of  the  training,  including  one  to  assess 
student  behaviors  that  correspond  with  thinking  dispositions.  This  connects  his  explicit  training 
approach  with  the  cultural  approach.  Glaser  expected  the  instructors  to  monitor  the  culture  of  the 
classroom  to  observe  the  day-to-day  behaviors  of  the  students. 

Teachers'  Ratings 

It  seemed  to  the  author  reasonable  to  expect  that  after  ten  weeks  of  special  work  in 
critical  thinking,  the  teachers  of  the  four  experimental  classes  ought  to  have  well-founded 
impressions  concerning  the  relative  abilities  manifested  by  their  students.  Consequently, 
a  few  days  after  the  retesting  had  been  completed,  a  mimeographed  sheet,  which  is 
reproduced  below,  was  sent  to  the  teachers  of  the  four  experimental  classes. 

BEHAVIOR  NO  1 

Does  he  select  significant  words  and  phrases  in  any  statement  that  is  important  to  him 
and  ask  that  they  be  carefully  defined?  That  is,  does  he  show  a  sensitivity  to  vague  or 
ambiguous  words  or  phrases  and  ask  just  what  is  meant  by  them?  For  example,  if 

someone  calls  another  person  a  "radical"  would  he  be  inclined  to  ask  what  is  meant  by  a 

1 62 

radical  -  how  can  one  tell  a  radical  from  a  non-radical? 

The  results  of  Glaser’s  experiment  were  positive,  showing  improvement  in  both  skills  and 
thinking  dispositions. 

There  is  evidence  for  some  students  of  improvement  in  general  disposition  to  consider 
problems  thoughtfully,  and  evidence  of  ability  to  think  more  critically  in  other  classes  in 
school,  at  home,  in  connection  with  personal  problems,  and  in  connection  with  speeches, 


160  Glaser,  E.M.  An  Experiment  in  the  Development  of  Critical  Thinking,  AMS  Press  Inc.,  New  York,  NY,  1941,  p. 
5. 

161  Glaser,  p.  99. 

162  Glaser,  p.  110. 
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advertisements,  and  arguments.  The  aspect  of  critical  thinking  which  appears  most 
susceptible  to  general  improvement  is  the  attitude  of  being  disposed  to  consider  in  a 
thoughtful  way  the  problems  and  subjects  that  come  within  the  range  of  one's  experience. 
An  attitude  of  wanting  evidence  for  beliefs  is  most  subject  to  general  transfer.  Develop¬ 
ment  of  skill  in  applying  the  methods  of  logical  inquiry  and  reasoning,  however,  appears 
to  be  specifically  related  to,  and  in  fact  limited  by,  the  acquisition  of  pertinent  knowledge 
and  facts  concerning  the  problem  or  subject  matter  toward  which  the  thinking  is  to  be 
directed.  There  is  no  reason  to  believe,  for  instance,  that  the  students  who  gained 
significantly  in  critical  thinking  as  measured  by  the  tests  are  now  appreciably  more 

163 

competent  to  make  critical  judgments  of  pieces  of  art,  music,  or  literature. 

Jennifer  Reed  conducted  an  experiment  to  teach  critical  thinking  skills  in  a  college  level  history 
class.  Although  her  experiment  focused  on  skills,  she  assessed  the  impact  of  her  teaching  on  both 
skills  and  thinking  dispositions.  Her  experiment  consisted  of: 

Four  sections  of  U.S.  History  1877  to  the  Present  participated  in  this  one  semester  study. 
Two  sections  were  randomly  selected  to  serve  as  the  experimental  group  and  the  other 
two  sections  served  as  the  control  group.  The  experimental  group  (n  =  29)  received 
approximately  90  minutes  of  explicit  instruction  distributed  over  the  semester  in  using 
Paul’s  model  for  critical  thinking  to  analyze  and  interpret  primary  source  documents.  In 
addition,  the  model  was  integrated  into  a  series  of  assigned  classroom  activities.  The 
control  group  (n  =  23)  was  taught  in  a  more  traditional  manner.164 

Students  did  improve  their  critical  thinking  skills,  but  Reed  found  little  impact  on  thinking 
dispositions: 

To  test  students’  dispositions  toward  critical  thinking,  students  in  both  groups  took  the 
California  Critical  Thinking  Dispositions  Inventory  during  the  first  two  weeks  of  the 
course  and  again  during  the  last  week  of  the  course.  Results  from  statistical  analyses  of 
the  scores  on  this  instrument  showed  no  significant  differences  between  the  experimental 
and  control  groups.  Further,  posttest  means  were  not  significantly  different  from  pretest 
means  in  either  group  (Pretest:  Experimental  M  =  296,  Control  M  =  297;  Posttest: 
Experimental  M  =  298,  Control  M  =  302).  It  appears  that  taking  a  single  history  course 
that  includes  explicit  instruction  in  Paul’s  model  but  does  not  emphasize  intellectual  traits 
of  the  critical  thinker  has  no  effect  on  students’  dispositions  toward  critical  thinking.165 

Monash  University  in  Australia  has  conducted  a  number  of  experiments  investigating  how  to 
teach  critical  thinking  as  part  of  The  Critical  Thinking  Study.  In  this  particular  experiment,  the 
researchers  investigated  whether  they  could  explicitly  teach  the  disposition  of  open-mindedness. 
Their  experiment  consisted  of: 

AOMT  [Actively  Open  Minded  Thinking]  Teaching  Strategies 

1.  Students  were  taught  about  some  of  the  empirical  evidence  for  myside  bias  and  the 
evidence  that  AOMT  reduces  bias  and  improves  thinking. 


Glaser,  p.  175. 

164  Reed,  J.H.  Effect  of  a  Model  for  Critical  Thinking  on  Student  Achievement  in  Primary  Source  Document  Analysis 
and  Interpretation,  Argumentative  Reasoning,  Critical  Thinking  Dispositions,  and  History  Content  in  a  Community 
College  History’  Course,  Ph.D.  dissertation.  University  of  South  Florida,  1998,  p.  xi. 

165  Reed,  p.  150. 
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2.  Exercises  that  focus  on  the  ability  of  students  to  find  alternative  explanations  or 
counter-evidence  for  a  given  claim. 

3.  Students  taught  that  good  arguments  must  take  into  account  all  the  relevant  evidence 
and  counter-arguments  or  possible  objections  to  the  reasoning  or  premises. 

4.  Exercises  in  which  students  must  criticise  arguments  in  support  of  their  own  position 
on  the  topic  under  discussion  and  suggest  evidence  or  arguments  against  their  position. 

5.  Exercises  in  which  students  are  instructed  not  just  to  pick  the  answer,  but  to  actively 
look  for  evidence  against  their  choice,  by  carefully  considering  the  alternatives.166 

To  determine  the  outcome  of  their  experiment,  the  researchers  administered  the  California 
Critical  Thinking  Skills  Test  (CCTST)  and  the  Graduate  Skills  Assessment  (both  focused  on 
thinking  skills)  and  Stanovich  and  West’s  Thinking  Dispositions  Questionnaire  (TDQ)  to  assess 
open-minded  thinking.  All  three  assessments  were  administered  before  and  after  the  training. 
The  researchers  found: 

Results 

Students  showed  no  significant  improvement  on  critical  thinking  tests  scores  on  either  the 
GSA  or  CCTST.  Students  showed  a  statistically  significant  improvement  in  open- 

i 

minded  attitudes,  as  measured  by  the  TDQ. 

Both  the  Reed  experiment  and  the  Monash  study  provide  evidence  for  the  differentiation 
between  thinking  skills  and  thinking  dispositions.  Teaching  explicitly  for  thinking  dispositions 
may  not  impact  skills  (Monash)  and  teaching  explicitly  for  skills  may  not  have  an  impact  on 
thinking  dispositions  (Reed). 

4.5.3.  Summary  -  Teaching  Thinking  Dispositions  Effectively 

It  is  clear  that  instruction  and  practice  can  strengthen  positive  thinking  dispositions.  Both 
aspects  of  training,  that  is,  creating  a  culture  that  expects  thinking  and  explicitly  teaching  for 
positive  thinking  dispositions,  appear  to  be  valuable.  An  individual  must  be  exposed  to  or  made 
aware  of  positive  thinking  dispositions  and  must  also  have  sufficient  opportunity  for  practice  in 
order  for  positive  thinking  dispositions  to  become  entrenched  and  habitual. 

Once  the  IC  identifies  and  confirms  the  thinking  dispositions  with  the  greatest  positive  impact, 
my  research  shows  that  it  is  possible  to  develop  training  that  instills  and  reinforces  the  thinking 
dispositions  needed  for  effective  intelligence  analysis. 

I  did  find  evidence  that  work,  classroom,  and  life  cultures  can  have  an  impact  on  thinking 
dispositions  and  that  explicitly  training  for  specific  thinking  dispositions  can  strengthen  them. 
However,  I  found  no  long-term  follow-up  that  assessed  whether  the  stronger  thinking 
dispositions  demonstrated  shortly  after  an  intervention  had  staying  power.  Did  the  individuals 
who  participated  in  these  training  programs  continue  to  have  the  disposition  to  engage  in 
effortful  thinking? 

The  impact  of  the  culture  on  an  individual’s  thinking  dispositions  also  points  to  the  importance 
of  the  work  environment  on  cognitive  performance.  Ensuring  a  work  environment  in  the  IC  that 

166  The  Monash  Critical  Thinking  Study,  Monash  University.  Available: 
http://arts.monash.edii.au/philosophv/research/thinking/Q7aomt.php.  Viewed:  April  12,  2010. 

Ih7  The  Monash  Critical  Thinking  Study,  Monash  University.  Available: 
http://arts.monash.edu.au/philosophy/research/thinking/07aomt.php.  Viewed:  April  12,  2010. 
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expects  and  emphasizes  good  thinking  will  reinforce  the  value  of  positive  thinking  dispositions 
and  might  ensure  that  positive  thinking  dispositions  do  have  staying  power. 
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5.0  CONCLUSIONS 

5.1  Findings 

The  primary  questions  guiding  this  research  were: 

What  is  the  value  and  feasibility  of  evaluating  a  candidate’s  thinking  dispositions  as  one 
component  of  intelligence  analyst  selection?  What  is  the  feasibility  of  training 
individuals  in  order  to  strengthen  their  positive  thinking  dispositions? 

Based  on  the  results  of  this  research,  I  believe  it  is  practical  and  valuable  to  use  thinking 
dispositions  as  one  component  of  intelligence  analyst  selection.  The  research  also  shows  that 
training  intelligence  analysts  to  improve  their  thinking  dispositions  can  have  a  positive  impact  on 
their  cognitive  activity. 

The  findings  that  contributed  to  these  conclusions  include: 

1 .  What  are  thinking  dispositions? 

a.  Thinking  dispositions  represent  an  individual’s  inclination  toward  thinking.  An 
individual’s  strength  in  positive  or  productive  thinking  dispositions  identifies  whether 
someone  is  likely  to  notice  when  effortful  thinking  is  required  and  the  degree  of 
inclination  to  make  the  effort  when  thinking  is  required.  This  description  is  accepted 
across  academic  disciplines  and  has  remained  consistent  over  time. 

b.  Thinking  dispositions  that  might  be  useful  for  intelligence  analysts  include: 

•  Enjoyment  of  cognitive  effort 

•  Intellectual  curiosity 

•  Open-mindedness 

•  Intellectual  honesty 

•  Tolerant  of  ambiguity 

•  Focus  on  evidence 

•  Persistent 

•  Systematic 

2.  How  do  thinking  dispositions  affect  cognitive  performance? 

a.  It  is  clear  that  thinking  dispositions  are  a  trigger  for  cognitive  activity.  When  an 
individual  is  disposed  to  think,  he  or  she  makes  the  effort  to  engage  cognitively. 

b.  The  specific  thinking  dispositions  possessed  by  an  individual  and  the  relative  strength 
of  individual  thinking  dispositions  influence  the  degree  to  which  an  individual  makes 
an  effort  to: 

•  Engage  cognitively  at  all 

•  Override  personal  beliefs  and  biases  to  ensure  an  open-minded  approach  to 
the  task 

•  Thoroughly  investigate  a  situation  in  order  to  find  relevant  infonnation  and  to 
look  at  all  sides  of  an  issue 

•  Avoid  premature  closure  of  the  thinking  task  and  exhibit  the  willingness  to 
tolerate  lack  of  a  clear  answer  (ambiguity)  while  investigation  continues 
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•  Reach  a  conclusion  only  after  finding  sufficient  evidence  to  support  the 
conclusion 

c.  Psychologists  who  have  conducted  experiments  into  cognitive  performance  have 
found  that  thinking  dispositions  can  predict  the  outcome  of  cognitive  perfonnance 
after  intellectual  ability  (IQ)  has  been  considered. 

3.  Can  thinking  dispositions  be  assessed?  Are  there  reliable  and  valid  methods  for  assessment? 

a.  Thinking  dispositions  can  be  assessed  and  a  variety  of  different  assessments  exist. 

4.  How  do  thinking  dispositions  relate  to  the  characteristics,  attributes,  and  traits  of  intelligence 
analysts? 

a.  Where  scholars  described  disposition-like  attributes  of  intelligence  analysts,  a 
significant  overlap  with  thinking  dispositions  does  exist.  The  thinking  dispositions 
we  have  identified  are  recognized  as  important  contributors  to  analytic  success. 

5.  How  are  an  individual’s  thinking  dispositions  established  or  subsequently  changed? 

a.  Thinking  dispositions  are  believed  to  be  developed  by  the  environment  and  culture  in 
which  an  individual  lives.  A  family  or  school  setting  can  foster  positive  or  negative 
thinking  dispositions  and  can  strengthen  or  weaken  thinking  dispositions  by  what  is 
modeled,  expected,  and  rewarded  in  the  culture  surrounding  the  thinker. 

b.  Positive  thinking  dispositions  can  be  strengthened  by  instruction  and  practice.  Both 
aspects  of  training,  that  is,  creating  a  culture  that  expects  effective  thinking  and 
explicitly  teaching  for  positive  thinking  dispositions,  appear  to  be  valuable. 

c.  One  critical  success  factor  appears  to  be  ensuring  sufficient  practice  for  the  positive 
thinking  dispositions  to  become  entrenched  and  habitual. 

d.  A  workplace  culture  can  encourage  and  strengthen  positive  thinking  dispositions  by 
expecting  cognitive  behaviors  and  outcomes  that  rely  on  positive  thinking 
dispositions. 

5.2  Recommendations 

In  order  to  gain  maximum  benefit  from  the  use  of  a  thinking  dispositions  assessment  in  the 

selection  of  analysts,  the  IC  should: 

1.  Conduct  a  comprehensive  job  analysis  to  identify  the  specific  traits  and  characteristics 
needed  by  intelligence  analysts.  Using  this  job  analysis,  identify  which  specific  thinking 
dispositions  have  the  most  positive  impact  on  analytic  performance. 

Numerous  authors  have  described  various  traits  and  capabilities  of  the  ideal  intelligence 
analyst.  These  traits  are,  however,  anecdotal.  The  IC  should  use  the  existing  intelligence 
literature  as  a  starting  point  for  developing  what  might  be  considered  a  definitive  description 
of  the  needed  competencies  and  traits  of  an  intelligence  analyst.  This  description  should 
include  thinking  dispositions. 

2.  Purchase,  adapt,  or  develop  assessments  that  measure  the  strength  of  an  individual’s  thinking 
dispositions. 

Once  an  understanding  of  analyst  competencies,  dispositions,  and  other  traits  exists,  develop 
an  assessment  that  addresses,  at  a  minimum,  the  thinking  dispositions  a  candidate  has.  The 
many  assessments  found  in  this  study  provide  a  starting  point  for  the  IC.  Although  not  all 
assessments  identified  in  this  study  were  found  to  be  equally  reliable  and  valid,  they  are  a 
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source  of  potentially  useful  questions  and  approaches  that  might  be  included  in  the  selection 
process  for  intelligence  analysts. 

3.  Validate  that  the  assessment  is  useful  in  identifying  individuals  with  thinking  dispositions 
that  have  a  positive  impact  on  intelligence  analysis. 

Because  of  the  potential  impact  of  using  a  thinking  dispositions  assessment  in  intelligence 
analyst  selection,  care  must  be  taken  to  ensure  that  the  assessment  tool  is  valid  and  useful  for 
selection.  Although  intelligence  analysis  does  not  lend  itself  to  quantitative  evaluation  such 
as  sales  quota  achievement,  it  is  possible  to  experimentally  assess  the  impact  of  thinking 
dispositions  on  an  individual’s  analytic  results.  The  experiments  conducted  by  Stanovich,  for 
example,  provide  a  model  for  how  this  might  be  done.  An  essential  ingredient  of  the 
experiment  would  be  a  standard  repertoire  of  intelligence  problems  and  databases  tailored  to 
the  participants’  specialties.  The  experiment  would  include  a  solution  for  each  problem  that 
represents  the  best  analysis  that  can  be  accomplished  given  the  information  in  the  databases. 
Experiment  facilitators  would  use  these  solutions  to  evaluate  each  participant’s  analysis  and 
would  be  able  to  compare  analytic  results  with  thinking  disposition  assessment  results. 

4.  Validate  that  the  assessment  is  fair  and  does  not  inappropriately  exclude  any  individual  or 
classes  of  individuals. 

Ensure  that  the  assessment  will  meet  the  “Uniform  Guidelines  on  Employee  Selection 
Procedures”  as  defined  by  the  Equal  Employment  Opportunity  Commission,  the  Civil 
Service  Commission,  the  Department  of  Labor,  and  the  Department  of  Justice.  These  can  be 
found  at  http://www.unifonnguidelines.com/unifonnguidelines.html. 

5.  Strengthen  the  thinking  dispositions  of  intelligence  analysts  already  at  work  in  the 
intelligence  community  by  improving  intelligence  analyst  training  programs.  The  goals  of 
these  improvement  efforts  are  to  ensure  that  the  training  culture  fosters  positive  thinking 
dispositions  and  includes  specific  training  on  thinking  dispositions. 

6.  Ensure  the  work  environment  continues  the  enculturation  process  through  its  efforts  to 
expect,  strengthen,  and  reward  strong  positive  thinking  dispositions. 
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APPENDIX  A  -  Thinking  Dispositions 


This  table  contains  lists  of  dispositions  found  during  the  search.  Source  indicates  the  original  source  of  the  definition  such  as  the  Foundation 
for  Critical  Thinking  or  the  APA  study  convened  by  Facione. 


Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

1.  seven  characterological  attributes  or  habits  of  mind...  truth-seeking,  open-mindedness,  analytic  ity, 
systematicity,  critical  thinking,  self  confidence,  inquisitive  ness,  and  maturity  of  judgment 

Moore 

LA. 

Critical 

2006 

Facione 

2.  According  to  Richard  Paul  and  Gerald  Nosich,  the  characteristics  of  critical  thinkers  include  thinking 
independently,  exercising  fair-mindedness,  developing  insight  into  egocentricity  and  sociocentricity, 
developing  intellectual  humility  and  suspending  judgment,  developing  intellectual  courage,  developing 
intellectual  good  faith  and  integrity,  developing  intellectual  perseverance,  developing  confidence  in 
reason,  exploring  thoughts  underlying  feelings  and  feelings  underlying  thoughts,  developing  intellectual 
curiosity. 

Moore 

LA 

Critical 

2006 

Foundation 

3.  inquisitiveness,  systematicity,  analyticity,  truth-seeking,  self-confidence,  and  maturity  (note  that  author 
left  off  “of  judgment”) 

Montgomery 

Medicine 

Critical 

2003 

Facione 

4.  "the  disposition  to  weigh  new  evidence  against  a  favored  belief  heavily  (or  lightly),  the  disposition  to 
spend  a  great  deal  of  time  (or  very  little)  on  a  problem  before  giving  up,  or  the  disposition  to  weigh 
heavily  the  opinions  of  others  in  forming  one's  own" 

Baron  (1985,1988)  has  called  such  tendencies  dispositions  toward  actively  open-minded  thinking. 
Overall,  the  subscales  in  this  thinking  dispositions  questionnaire  measure  the  following  dimensions: 
epistemological  absolutism,  willingness  to  perspective-switch,  willingness  to  decontextualize,  and  the 
tendency  to  consider  alternative  opinions  and  evidence 

Stanovich 

psychology 

Rational 

2004 

Baron 

5.  Need  for  cognition  -  a  dispositional  variable  reflecting  the  tendency  toward  thoughtful  analysis  and 
reflective  thinking 

Stanovich 

psychology 

Rational 

2004 

Cacioppi 

6.  Overall,  the  subscales  in  this  thinking  dispositions  measure  the  following  dimensions:  epistemological 
absolutism,  willingness  to  perspective-switch,  willingness  to  decontextualize,  and  the  tendency  to 
consider  alternative  opinions  and  evidence.  Established  subscales  were  included  (e.g.,  Rokeach,  1960) 
as  well  as  subscales  new  to  the  literature  (such  as  an  activity  open-minded  thinking  scale  based  on 

Baron's  1985a,  1988,  work). 

Stanovich 

psychology 

Rational 

1999 

Stanovich  and 
various  -  he 
listed 
individual 
authors  with 
some  of  the 
individual 
dispositions 

7.  Perhaps  the  strongest  similarities  were  with  the  two  dispositional  factors  that  Schommer  (1990,  1993) 

Stanovich 

psychology 

Rational 

1999 

Schommer 
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Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

called  "belief  in  simple  knowledge"  and  "belief  in  certain  knowledge." 

8.  Flexible  Thinking  Scale,  Openness-Ideas,  Openness- Values,  Absolutism  Scale,  Dogmatism  Scale, 
Categorical  Thinking,  (TDC  Composite  of  the  previous  items),  Superstitious  Thinking,  Counterfactual 
Thinking,  Social  Desirability  Scale 

Stanovich 

psychology 

Rational 

1999 

Stanovich  and 
various  -  he 
listed 
individual 
author  with 
some  of  the 
individual 
dispositions 

9.  Willingness  to  postpone  closure 

Willingness  to  consider  alternative  opinions 

Willingness  to  consider  contradictory  evidence 

Openness 

Disposition  toward  absolutism 

Disposition  toward  dogmatism 

Disposition  toward  categorical  thinking 

Disposition  toward  counterfactual  thinking 

Disposition  toward  belief  identification 

Need  for  cognition 

Stanovich 

psychology 

Rational 

1999 

Stanovich  and 
various  -  he 
listed 
individual 
author  with 
some  of  the 
individual 
dispositions 

10.  Reflectivity/impulsivity,  need  for  cognition,  vigilance,  categorical  thinking,  flexible  thinking, 
counterfactual  thinking,  and  actively  open-minded  thinking 

Stanovich  &  West 
Psychology 

Rational 

2000 

Stanovich  & 
West 

11.  actively  openminded  thinking,  absolutism,  dogmatism,  counterfactual  thinking,  and  belief  in  the 
paranormal 

Newstead,  et  al 
Psychology 

Rational 

2004 

Stanovich  & 
West 

12.  Actively  open  minded  thinking,  counterfactual  thinking,  absolutism,  dogmatism,  paranormal  beliefs, 
social  desirability  response 

Together  they  made  up  what  he  called  the  thinking  dispositions  composite 

Stanovich  and 

West 

Psychology 

Rational 

1998 

Stanovich 

13.  Cognitive  confidence,  reflectivity,  need  for  cognition,  ideational  generativity,  dispositions  toward 
confirmation  bias  and  premature  closure,  etc. 

Stanovich  and 

West 

Psychology 

Rational 

1998 

Oatley  and 
Overton 

14.  Flexible  thinking,  openness  -  ideas,  openness  -  values,  absolutism,  dogmatism,  categorical  thinking, 
superstitious  thinking,  counterfactual  thinking,  outcome  bias  (indicator  of  disposition  for 
decontextualization),  social  desirability  response  bias. 

Composite  actively  open-minded  thinking  consist  of  summing  the  scores  on  the  Flexible  Thinking, 
Openness-Ideas,  and  Openness-Values  scales  and  subtracting  the  sum  of  the  Absolutism,  Dogmatism, 

Stanovich  and 

West 

Psychology 

Rational 
and  critical 

1997 

Stanovich 
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Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

and  Categorical  Thinking  scales. 

15.  1.  The  disposition  to  be  broad  and  adventurous :  the  tendency  to  be  open-minded,  to  explore  alternative 
views;  an  alertness  to  narrow  thinking;  the  ability  to  generate  multiple  options. 

2.  The  disposition  toward  sustained  intellectual  curiosity,  the  tendency  to  wonder,  probe,  find 
problems;  a  zest  for  inquiry;  an  alertness  for  anomalies;  the  ability  to  observe  closely  and  formulate 
questions. 

3.  The  disposition  to  clarify  and  seek  understanding :  a  desire  to  understand  clearly,  to  seek  connections 
and  explanations;  an  alertness  to  unclarity  and  need  for  focus;  an  ability  to  build  conceptualizations. 

4.  The  disposition  to  be  planful  and  strategic,  the  drive  to  set  goals,  make  and  execute  plans,  envision 
outcomes;  alertness  to  lack  of  direction;  the  ability  to  formulate  goals  and  plans. 

5.  The  disposition  to  be  intellectually  careful,  the  urge  for  precision,  organization,  thoroughness;  an 
alertness  to  possible  error  or  inaccuracy;  the  ability  to  process  information  precisely. 

6.  The  disposition  to  seek  and  evaluate  reasons:  the  tendency  to  question  the  given,  to  demand 
juslification;  an  alertness  to  the  need  for  evidence;  the  ability  to  weigh  and  assess  reasons. 

7.  The  disposition  be  metacognitive:  the  tendency  to  be  aware  of  and  monitor  the  flow  of  one's  own 
thinking;  alertness  to  complex  thinking  situations;  the  ability  to  exercise  control  of  mental  processes 
and  be  reflective. 

Tishman,  Jay, 
Perkins 

Education 

Good 

thinking 

Summ 

er 

1993 

Tishman,  Jay, 
Perkins 

16.  Introduces  the  concept  that  thinking  dispositions  are  made  of  three  attributes:  inclination,  sensitivity 
and  abilities  then  identifies  these  dispositions: 

1.  The  disposition  to  be  broad  and  adventurous. 

Kev  inclinations:  The  tendencv  to  be  onen-minded  and  to  look  bevond  what  is  given;  the  imnulse  to 
probe  assumptions  and  examine  alternative  points  of  view;  the  desire  to  tinker  with  boundaries  and  play 
with  new  ideas;  the  urge  to  speculate,  generate  many  options,  and  explore  multiple  interpretations. 

Kev  sensitivities:  An  alertness  to  binariness.  dogmatism,  sweening  generalities,  narrow  thinking, 
parochialism,  and  occasions  when  alternative  perspectives  are  neglected. 

Kev  abilities:  The  ability  to  identify  assumptions,  to  look  at  things  from  other  points  of  view,  to 
generate  and  review  multiple  options;  brainstorming;  empathic  thinking;  flexible  thinking. 

2.  The  disposition  toward  sustained  intellectual  curiosity. 

Kev  inclinations:  A  zest  for  inauirv;  the  urge  to  find  and  pose  problems;  the  tendencv  to  wonder, 
question,  probe. 

Kev  sensitivities:  An  alertness  to  unasked  questions,  anomalies,  hidden  facets;  detection  of  gaps  in  one's 
knowledge  or  understanding;  noticing  what  is  unknown  or  unclear. 

Kev  abilities:  The  ability  to  observe  closelv.  to  identify  and  challenge  assumptions,  to  formulate  and 
investigate  provocative  questions,  to  focus  and  persist  in  a  line  of  inquiry. 

3.  The  disposition  to  clarify  and  seek  understanding. 

Tishman,  Jay, 
Perkins 

Education 

Good 

thinking 

Jan. 

1993 

Tishman,  Jay, 
Perkins 
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Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

Kev  inclinations:  A  desire  to  annrehend  things  clearly;  the  impulse  to  anchor  ideas  to  experience  and 
seek  connections  to  prior  knowledge;  an  urge  to  sharpen  conceptions  and  examples;  a  desire  to  grasp 
the  essence  of  things. 

Kev  sensitivities:  Alertness  to  unclaritv  and  discomfort  with  vagueness;  alertness  to  superficiality; 
detection  of  occasions  needing  a  sharper  focus;  a  leaning  toward  hard  questions. 

Kev  abilities:  The  ability  to  ask  pointed  auestions  and  to  build  complex  conceptualizations;  the  ability 
to  apply  and  exemplify  ideas,  to  make  analogies  and  comparisons,  to  identify  and  classify  details. 

4.  The  disposition  to  be  planful  and  strategic. 

Kev  inclinations:  The  urge  to  set  goals  and  to  make  and  execute  plans;  the  tendency  to  approach  things 
in  a  calculated  and/or  stepwise  fashion;  a  desire  to  think  ahead. 

Kev  sensitivities:  Alertness  to  aimlessness,  lack  of  direction,  lack  of  orientation;  alertness  to  off-hand 
thinking  and  sprawling  thinking. 

Kev  abilities:  The  ability  to  formulate  goals  and  to  evaluate  alternative  modes  of  approach;  the  ability  to 
make  and  execute  plans  and  to  forecast  possible  outcomes. 

5.  The  disposition  to  be  intellectually  careful. 

Kev  inclinations:  The  urge  for  precision;  a  hunger  for  mental  orderliness  and  organization;  a  desire  to 
be  thorough. 

Kev  sensitivities:  Alertness  to  the  possibility  of  error,  to  disorder  and  disorganization;  awareness  of  the 
abiding  potential  for  inaccuracy  and  inconsistency. 

Kev  abilities:  The  ability  to  process  information  precisely,  to  recognize  and  apply  intellectual  standards, 
to  construct  order  out  of  disarray. 

6.  The  disposition  to  seek  and  evaluate  reasons. 

Kev  inclinations:  A  leaning  toward  healthy  skepticism;  the  tendency  to  auestion  the  given,  to  probe 
assumptions  and  biases;  the  drive  to  pursue  and  demand  justification;  the  urge  to  discover  underlying 
grounds  and  sources. 

Kev  sensitivities:  an  alertness  to  evidential  foundations;  a  responsiveness  to  superficiality  and  over- 
generalization,  a  wariness  of  gaps  in  knowledge. 

Kev  abilities:  The  ability  to  distinguish  cause  and  effect,  the  ability  to  identify  logical  structure;  the 
ability  to  reason  inductively,  the  ability  to  weigh  and  assess  reasons. 

7.  The  disposition  to  be  metacognitive. 

Kev  inclinations:  The  urge  to  be  cognitivelv  self-aware  and  to  monitor  the  flow  of  one's  thinking;  the 
impulse  to  stand  back  and  take  stock;  the  desire  to  be  self-challenging. 

Kev  sensitivities:  Alertness  to  loss  of  control  of  one's  thinking;  detection  of  complex  thinking  situations 
reauiring  self-monitoring;  recognition  of  the  need  to  look  back  on  a  thinking  episode. 

Kev  abilities:  The  ability  to  exercise  executive  control  of  mental  processes,  to  conceive  of  the  mind  as 
active  and  interpretive,  to  be  self-evaluative,  to  reflect  on  prior  thinking. 

17.  Examples  of  some  thinking  dispositions  that  have  been  investigated  by  psychologists  are:  actively  open- 

Stanovich 

Rational 

2009 

Not  stated 
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Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

minded  thinking,  need  for  cognition  (the  tendency  to  think  a  lot),  consideration  of  future  consequences, 
need  for  closure,  superstitious  thinking,  and  dogmatism 

Psychology 

18.  the  tendency  to  collect  information  before  making  up  one's  mind,  the  tendency  to  seek  various  points  of 
view  before  coming  to  a  conclusion,  the  disposition  to  think  extensively  about  a  problem  before 
responding,  the  tendency  to  calibrate  the  degree  of  strength  of  one's  opinion  to  the  degree  of  evidence 
available,  the  tendency  to  think  about  future  consequences  before  taking  action,  the  tendency  to 
explicitly  weigh  pluses  and  minuses  of  situations  before  making  a  decision,  and  the  tendency  to  seek 
nuance  and  avoid  absolutism 

Stanovich 

Psychology 

Rational 

2009 

Not  stated 

19.  Provides  another  list  in  the  same  book.  This  is  by  way  of  an  example  of  Stanovich’s  estimate  of  G. 
Bush’s  thinking  dispositions: 

overconfident;  low  in  typical  intellectual  engagement;  low  in  openness  to  experience;  high  in  belief 
perseverance;  high  in  confirmation  bias;  high  in  faith  in  intuition;  high  in  impulsiveness;  high  in 
one-sided  thinking;  low  in  need  for  cognition;  low  in  openness  to  experience;  does  not  engage  in 
counterfactual  thinking;  treats  beliefs  as  possessions  (has  high  belief  identification);  is  high  in  need 
for  closure,  and  low  in  thought  flexibility. 

Stanovich 

Psychology 

Rational 

2009 

Not  stated 

20.  Focused  on  two  dispositions  as  they  relate  to  critical  thinking  in  heuristics  and  bias:  need  for  cognition 
and  actively  openminded  thinking 

Stanovich,  West, 
Toplak 

Psychology 

Critical 

undate 

d 

Not  stated 

21.  (1)  truth-seeking,  (2)  open-mindedness,  (3)  analyticity,  (4)  systematicity,  (5)  self-confidence,  (6) 
inquisitiveness,  and  (7)  maturity. 

Perkins 

Education 

intelligence 

1995 

Facione 

22.  (1)  broad  and  adventurous  thinking,  (2)  sustained  intellectual  curiosity,  (3)  clarifying  and  seeking 
understanding,  (4)  being  planful  and  strategic,  (5)  being  intellectually  careful,  (6)  seeking  and 
evaluating  reasons,  and  (7)  metacognitive  self-management  (monitoring  and  guiding  your  own 
thinking). 

Perkins 

Education 

intelligence 

1995 

Tishman,  Jay, 
Perkins 

23.  Perkins  variation  on  the  Tishman  et  al  list: 

Clear.  The  disposition  toward  thinking  that  is  clear,  coherent,  precise,  specific,  and  well-organized. 
Broad.  The  disposition  toward  thinking  that  is  broad,  adventurous,  flexible,  independent.  Respect  for 
and  appreciation  of  other  perspectives  through  tolerance,  openmindedness,  and  empathy.  The 
disposition  to  seek  and  find  connections. 

Deep.  The  disposition  to  understand  deeply,  to  seek  underlying  unities  in  the  form  of  laws,  theories, 
frameworks  and  principles,  to  fathom  the  causes  and  other  governing  factors  of  ideas,  things,  and 
events. 

Sound.  The  disposition  toward  thinking  that  is  accurate,  thorough,  fair,  knowledgeable,  logical,  and 
well-supported  by  evidence.  Concern  with  truth,  judiciousness,  relevance,  and  functionality,  in  the 

Perkins 

Education 

intelligence 

1995 

Perkins 
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Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

sense  of  the  functionality  of  a  sound  design. 

Curious.  The  disposition  toward  thinking  that  is  curious,  questioning,  probing,  inquisitive.  The 
disposition  to  find  out  about  things  and  to  persist  in  inquiry. 

Strategic.  The  disposition  toward  thinking  that  is  strategic  and  planful  in  style,  and  that  uses  stock 
thinking  strategies,  graphic  organizers,  and  other  devices  to  sustain  effective  organization. 

Aware.  The  disposition  toward  thinking  that  is  metacognitive,  aware  of  itself,  critical  of  its  own  patterns 
and  progress,  and  that  edits  its  own  practices.  This  includes  awareness  of  affective  dimensions  of 
thinking — the  clues  feelings  provide  us  with,  how  emotions  stimulate  or  stifle  a  link  of  thought 

24.  From  Perkins 

1.  The  disposition  to  be  broad  and  adventurous 

2.  The  disposition  toward  wondering,  problem  finding,  and  investigating 

3.  The  disposition  to  build  explanations  and  understandings 

4.  The  disposition  to  make  plans  and  be  strategic 

5.  The  disposition  to  be  intellectually  careful 

6.  The  disposition  to  seek  and  evaluate  reasons 

7.  The  disposition  to  be  metacognitive 

Perkins 

Education 

Good 

2004 

Tishman,  Jay, 
Perkins 

25.  open-mindedness,  inquisitiveness,  systematic ity,  analyticity,  truth-seeking,  critical  thinking  self- 
confidence,  and  maturity 

Perkins 

Education 

Good 

2004 

Facione 

26.  Critical  Thinkers  ... 

Are  honest  with  themselves,  acknowledging  what  they  don't  know,  recognizing  their  limitations,  and 
being  watchful  of  their  own  errors. 

Regard  problems  and  controversial  issues  as  exciting  challenges. 

Strive  for  understanding,  keep  curiosity  alive,  remain  patient  with  complexity,  and  are  ready  to  invest 
time  to  overcome  confusion. 

Base  judgments  on  evidence  rather  than  personal  preferences,  deferring  judgment  whenever  evidence  is 
insufficient.  They  revise  judgments  when  new  evidence  reveals  error. 

Are  interested  in  other  people's  ideas  and  so  are  willing  to  read  and  listen  attentively,  even  when  they 
tend  to  disagree  with  the  other  person. 

Recognize  that  extreme  views  (whether  conservative  or  liberal)  are  seldom  correct,  so  they  avoid  them, 
practice  fairmindedness,  and  seek  a  balanced  view. 

Practice  restraint,  controlling  their  feelings  rather  than  being  controlled  by  them,  and  thinking  before 
acting. 

Ruggerio 

Humanities 

Critical 

2003 

Ruggerio 

27.  From  Wikipedia  -  Habits  of  mind 

Wikipedia 

Critical  and 

2009 

Costa 
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1 .  Persisting  -  Stick  to  it. 

2.  Thinking  and  communicating  with  clarity  and  precision  -  Be  clear. 

3.  Managing  impulsivity  -  Take  your  time. 

4.  Gathering  data  through  all  senses  -  Use  your  natural  pathways. 

5.  Listening  with  understanding  and  empathy  -  Understand  others. 

6.  Creating,  imagining,  innovating  -  Try  a  different  way. 

7.  Thinking  flexibly  -  Look  at  it  another  way. 

8.  Responding  with  wonderment  and  awe  -  Have  fun  figuring  it  out. 

9.  Thinking  about  your  thinking  (metacognition)  -  Know  your  knowing. 

10.  Taking  responsible  risks  -  Venture  out. 

1 1 .  Striving  for  accuracy  and  precision  -  Check  it  again. 

12.  Finding  humour  -  Laugh  a  little. 

13.  Questioning  and  problem  posing  -  How  do  you  know. 

14.  Thinking  interdependently  -  Work  together. 

15.  Applying  past  knowledge  to  new  situations  -  Use  what  you  learn. 

16.  Remaining  open  to  continuous  learning  -  Learn  from  experiences. 

creative 

28.  Sternberg’s  Mental  Self  Government  Thinking  Stvle  profiles: 

three  functions  of  mental  self-government  are  legislative,  executive,  and  judicial 
four  forms  of  mental  self-government:  monarchic,  hierarchic,  oligarchic,  and  anarchic 

Two  levels  are  local  and  global 

Two  scopes  are  internal  and  external 

Two  leanings  are  liberal  and  conservative 

Black,  McCoach 
Education 

Psychiatry 

None 

specified 

Sternberg 

29.  approaches  to  life  and  living  which  characterize  critical  thinking  include: 

*  inquisitiveness  with  regard  to  a  wide  range  of  issues, 

*  concern  to  become  and  remain  well-informed, 

*  alertness  to  opportunities  to  use  critical  thinking, 

*  trust  in  the  processes  of  reasoned  inquiry, 

*  self-confidence  in  one’s  own  abilities  to  reason, 

*  open-mindedness  regarding  divergent  world  views, 

*  flexibility  in  considering  alternatives  and  opinions 

*  understanding  of  the  opinions  of  other  people, 

*  fair-mindedness  in  appraising  reasoning, 

*  honesty  in  facing  one’s  own  biases,  prejudices,  stereotypes,  or  egocentric  tendencies, 

*  prudence  in  suspending,  making  or  altering  judgments, 

Facione 

Philosophy 

Critical 

2009 

APA 
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*  willingness  to  reconsider  and  revise  views  where  honest  reflection  suggests  that  change  is  warranted. 

30.  Characteristics  of  critical  thinkers: 

*  clarity  in  stating  the  question  or  concern, 

*  orderliness  in  working  with  complexity, 

*  diligence  in  seeking  relevant  information, 

*  reasonableness  in  selecting  and  applying  criteria, 

*  care  in  focusing  attention  on  the  concern  at  hand, 

*  persistence  though  difficulties  are  encountered, 

*  precision  to  the  degree  permitted  by  the  subject  and  the  circumstances. 

Facione 

Philosophy 

Critical 

2009 

APA 

31.  From  the  expert  consensus  description  of  the  ideal  critical  thinker 

The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  openminded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  in  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  the  circumstances  of  inquiry  permit. 

Facione 

Philosophy 

Critical 

2009 

APA 

32.  Facione’s  list  of  dispositions: 

Inquisitive,  systematic,  judicious,  truthseeking,  analytical,  openminded,  confident  in  reasoning 

Facione 

Philosophy 

Critical 

2009 

Facione 

33.  From  a  web  site  called  habits  of  the  mind: 

The  16  Habits  of  Mind  identified  by  Costa  and  Kallick  include: 

Persisting 

Thinking  and  communicating  with  clarity  and  precision 

Managing  impulsivity 

Gathering  data  through  all  senses 

Listening  with  understanding  and  empathy 

Creating,  imagining,  innovating 

Thinking  flexibly 

Responding  with  wonderment  and  awe 

Thinking  about  thinking  (metacognition) 

Taking  responsible  risks 

Striving  for  accuracy 

F  inding  humor 

Questioning  and  posing  problems 

Thinking  interdependently 

Website  Habits  of 
the  Mind 

strategic 

reasoning, 

insightful¬ 

ness, 

persever¬ 

ance, 

creativity, 

and 

craftsman¬ 

ship 

2001 

Costa  and 
Killick 
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Applying  past  knowledge  to  new  situations 

Remaining  open  to  continuous  learning 

34.  From  the  expert  consensus  description  of  the  ideal  critical  thinker: 

The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  openminded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  in  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  the  circumstances  of  inquiry  permit. 

Facione 

Philosophy 

Critical 

2000 

APA 

35.  truth-seeking,  open-mindedness,  analyticity,  systematicity,  CT  self-confidence,  inquisitiveness,  and 
maturity  of  judgment 

Facione 

Philosophy 

Critical 

2000 

Facione 

36.  Critical  thinkers: 

1.  seek  a  statement  of  the  thesis  or  question; 

2.  seek  reasons; 

3.  try  to  be  well  informed; 

4.  use  credible  sources  and  mention  them; 

5.  take  into  account  the  total  situation; 

6.  keep  their  thinking  relevant  to  the  main  point; 

7.  keep  in  mind  the  original  or  most  basic  concern; 

8.  look  for  alternatives; 

9.  are  open-minded  and 

a.  seriously  consider  points  of  view  other  than  their  own; 

b.  reason  from  starting  points  with  which  they  disagree  without  letting  the  disagreement 
interfere  with  their  reasoning; 

c.  withhold  judgment  when  the  evidence  and  reasons  are  insufficient 

10.  take  a  position  and  change  a  position  when  the  evidence  and  reasons  are  sufficient  to  do  so; 

1 1 .  seek  as  much  precision  as  the  subject  permits; 

12.  deal  in  an  orderly  manner  with  the  parts  of  a  complex  whole; 

13.  employ  their  critical  thinking  abilities; 

14.  are  sensitive  to  the  feelings,  level  of  knowledge,  and  degree  of  sophistication  of  others. 

Ennis  &  Norris 
Philosophy 

Critical 

1996 

Ennis  &  Norris 

37.  Selected  thinking  dispositions: 

1 .  Seek  a  clear  statement  of  a  problem,  a  thesis,  a  question. 

2.  Deliberately  examine  a  variety  of  viewpoints. 

3.  Seek  to  be  well  informed. 

4.  Use  credible  sources. 

5.  Seek  a  number  of  alternatives. 

Beyer 

Education 

Skillful 

Effective 

functional 

good 

1987 

Ennis  and 
others 
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6.  Seek/give  reasons. 

7.  Seek/provide  evidence. 

8.  Be  open-minded. 

9.  Be  willing  to  change  a  position/judgment  when  evidence  and  reasons  are  sufficient  to  do  so. 

10.  Judge  in  terms  of  situations,  issues,  purposes,  and  consequences  (not  in  terms  of  fixed,  dogmatic 
precepts  or  emotional,  wistful  thinking). 

11.  Persist  in  carrying  out  a  thinking  task. 

12.  Be  slow  to  believe — be  skeptical. 

13.  Be  objective. 

14.  Suspend  judgment  when  appropriate/sufficient  evidence  and  reasoning  are  lacking. 

38.  Educator  Arthur  L.  Costa  has  suggested  a  list  of  behaviors  that  he  believes  are  indicative  of  an  effective 
thinker.  Such  an  individual,  he  writes: 

1.  Persists  in  a  thinking  task,  applying  alternative  methods  until  a  goal  is  achieved. 

2.  Deliberately  plans  how  to  execute  a  thinking  task  by  clarifying  goals,  identifying  givens,  and 
carefully  selecting  methods  and  data. 

3.  Exhibits  flexibility  in  thinking,  approaching  a  task  from  a  number  of  perspectives  or  angles. 

4.  Tells  the  steps  engaged  in  when  executing  a  thinking  skill  or  strategy. 

5.  Identifies  missing  data  in  a  problem-solving  situation  and  how  to  locate  it. 

6.  Goes  over  test  answers,  papers,  and  reports  to  check  for  accuracy,  completeness,  and  clarity. 

7.  Recognizes  discrepancies  in  the  environment  and  raises  questions  about  them. 

8.  Asks  his/her  own  questions  about  causation,  relationships,  hypothetical  situations,  and  stimuli. 

9.  Gives  and  requests  evidence  and  reasoning  in  support  of  assertions. 

10.  Draws  on  past  experience  and  knowledge  and  the  accumulative  knowledge  of  others. 

11.  Applies  knowledge  and  skills  learned  in  one  context  to  another  context. 

12.  Uses  and  insists  on  precise  language. 

13.  Expresses  enjoyment  in  thinking. 

14.  Expresses  pride  in  the  way  in  which  he/she  goes  about  thinking  and  the  results  he/she  gets. 

Beyer 

Education 

Effective 

1987 

Costa 

39.  good  thinkers  are  characteristically  curious,  open-minded,  persistent,  and  willing  to  take  risks 

Kirby  & 

Kuykendall 

Education 

good 

1991 

Wiggins 

40.  Attitudes  or  habits  of  mind  -  These  habits,  commitments  and  sensitivities  include  such  things  as: 

•  open-mindedness  (willingness  to  withhold  judgment  and  seek  new  evidence  or  points  of  view  when 
existing  evidence  is  inadequate  or  contentious,  and  willingness  to  consider  evidence  against  one's  view 

Bailin 

Education 

Critical 

1993 

Bailin 
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and  to  revise  one's  view  should  the  evidence  warrant  it); 

•  fair-mindedness  (willingness  to  give  fair  consideration  to  alternative  points  of  view  and  commitment 
to  open,  critical  discussion  of  theories,  practices  and  policies  where  all  views  are  given  a  fair  hearing); 

•  desire  for  truth  (commitment  to  having  justified  beliefs,  values  and  actions); 

•  independent-mindedness  (the  willingness  and  personal  strength  to  stand  up  for  one's  firmly  held 
beliefs); 

•  an  inquiring  attitude  (an  inclination  to  question  the  clarity  and  support  for  claims  one  is  asked  to 
accept); 

•  respect  for  high  quality  products  and  performances  (appreciation  of  good  design  and  effective 
performance); 

•  an  intellectual  work  ethic  (commitment  to  carrying  out  relevant  thinking  tasks  in  a  competent 
manner). 

4 1 .  The  critical  spirit,  which  Siegel  views  as  being  of  equal  importance  with  the  reason  assessment 
component,  indicates  that  the  thinker  values  good  reasons  and  is  disposed  to  assess  reasons  and  to 
govern  beliefs  and  actions  on  the  basis  of  such  assessment. 

Bailin 

Education 

Critical 

1997 

Siegel 

42.  Ennis  also  includes  a  list  of  tendencies  or  dispositions  in  his  conception  of  critical  thinking  which 

includes:  the  disposition  to  seek  a  clear  statement  of  the  statement  or  question,  to  seek  reasons,  to  try  to 
be  well-informed,  to  use  credible  sources  and  mention  them,  to  take  into  account  the  total  situation,  to 
try  to  remain  relevant  to  the  main  point,  to  keep  in  mind  the  original  or  basic  concern,  to  look  for 
alternatives,  to  be  open-minded,  to  take  a  position  when  the  evidence  and  reasons  are  sufficient,  to  seek 
as  much  precision  as  the  subject  permits,  to  deal  in  an  orderly  manner  with  the  parts  of  a  complex 
whole,  and  to  be  sensitive  to  the  feelings,  level  of  knowledge,  and  degree  of  sophistication  of  others. 

Bailin 

Education 

Critical 

1997 

Ennis 

43.  Ennis  (1987)  sees  dispositions  as  including  a  person's  willingness  to  be  open-minded,  well-informed, 
change  positions  when  the  evidence  warrants,  stick  to  the  point,  and  be  sensitive  to  the  feelings,  level  of 
knowledge  and  degree  of  sophistication  of  others. 

Arter 

Education 

Higher 

order 

thinking 

1987 

Ennis 

44.  Similarly,  Paul  (1986)  considers  "strong  sense  critical  thinking"  to  include  the  willingness  not  only  to 
reason  but  also  to  examine  one's  whole  frame  of  reference  and  belief  system. 

Arter 

Education 

Higher 

order 

1987 

Paul 

45.  A.  Dispositions 

1.  Seek  a  clear  statement  of  the  thesis  or  question 

2.  Seek  reasons 

3.  Try  to  be  well  informed 

4.  Use  and  mention  credible  sources 

5.  Take  into  account  the  total  situation 

Ennis 

Education 

Critical 

1987 

Ennis 
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6.  Try  to  remain  relevant  to  the  main  point 

7.  Keep  in  mind  the  original  and/or  basic  concern 

8.  Look  for  alternatives 

9.  Be  open-minded 

a)  Consider  seriously  other  points  of  view  than  one's  own  (dialogical  thinking) 

b)  Reason  from  premises  with  which  one  disagrees — without  letting  the  disagreement  interfere 
with  one's  reasoning  (suppositional  thinking) 

c)  Withhold  judgment  when  the  evidence  and  reasons  are  insufficient 

10.  Take  a  position  (and  change  a  position)  when  the  evidence  and  reasons  are  sufficient  to  do  so 

11.  Seek  as  much  precision  as  the  subject  permits 

12.  Deal  in  an  orderly  manner  with  the  parts  of  a  complex  whole 

13.  Use  one's  critical  thinking  abilities 

14.  Be  sensitive  to  the  feelings,  level  of  knowledge,  and  degree  of  sophistication  of  others 

46.  Characteristics  of  a  good  thinker  (edited  by  the  author  to  focus  on  dispositions): 
listens  carefully  to  other  people's  ideas; 

understands  the  difference  between  winning  an  argument  and  being  right; 

recognizes  that  most  real-world  problems  have  more  than  one  possible  solution  and  that  those  solutions 
may  differ  in  numerous  respects  and  may  be  difficult  to  compare  in  terms  of  a  single  figure  of  merit; 
looks  for  unusual  approaches  to  complex  problems; 

can  strip  a  verbal  argument  of  irrelevancies  and  phrase  it  in  terms  of  its  essentials; 

understands  the  differences  among  conclusions,  assumptions,  and  hypotheses; 

habitually  questions  one's  own  views  and  attempts  to  understand  both  the  assumptions  that  are  critical 

to  those  views  and  the  implications  of  the  views; 

is  sensitive  to  the  difference  between  the  validity  of  a  belief  and  the  intensity  with  which  it  is  held; 
can  represent  differing  viewpoints  without  distortion,  exaggeration,  or  caricaturization; 
is  aware  of  the  fact  that  one's  understanding  is  always  limited,  often  much  more  so  than  would  be 
apparent  to  one  with  a  noninquiring  attitude;  and 

recognizes  the  fallibility  of  one's  own  opinions,  the  probability  of  bias  in  those  opinions,  and  the  danger 

Nickerson 

education 

Good 

1987 

Nickerson 

47.  To  wonder 

To  be  critical 

To  respect  others 

To  be  inventive 

To  seek  alternatives 

Lipman 

Education 

No 

descriptor 

1987 

Lipman 

98 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-20 14-2455;  Cleared  21  May  2014 


Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

To  be  inquisitive 

To  care  for  the  tools  of  inquiry 

To  cooperate  intellectually 

To  be  committed  to  self  corrective  method 

To  feel  a  need  for  principles,  ideals,  reasons,  and  explanations 

48.  These  lists  of  attitudes  and  dispositions  that  accompany  good  thinking  generally  include  being 

reflective,  openminded,  curious,  persevering,  precise,  able  to  tolerate  ambiguity,  and  so  forth.  Some 
sources  also  include  internal  locus  of  control  and  self-esteem  as  attitudes  accompanying  effective 
thinking. 

JB  Baron 

Education 

No 

descriptor 

1987 

various 

49.  the  habits  of  mind  as  intellectual  integrity,  open-mindedness,  mental  alertness,  systematic  diligence, 
confidence  in  the  use  of  reason,  intellectual  curiosity,  and  prudent  maturity  of  judgment 

Facione 

General 

critical 

1997 

Facione 

50.  In  their  positive  manifestation,  these  seven  characterological  attributes  or  habits  of  mind,  if  you  prefer, 
are  named  truth-seeking,  open-mindedness,  analyticity,  systematicity,  critical  thinking  self-confidence, 
inquisitiveness,  and  maturity  of  judgment 

Facione 

General 

critical 

1997 

Facione 

51.  List  of  items  assessed  by  CCTDI: 

Truthseeking 

Open-mindedness 

Analyticity 

Systematicity 

Critical  Thinking  Self-Confidence 

Inquisitiveness 

Maturity  of  Judgment 

Insight  Assessment 
General 

critical 

2009 

Facione 

52.  need  for  cognition  and  need  for  epistemic  closure 

Hofer 

psychology 

No 

descriptor 

2004 

Caccioppo, 

Petty, 

Feinstein, 

Jarvis, 

Kruglanski 

53.  The  competencies  focusing  on  the  intellectual  traits,  virtues  or  dispositions 

Standard  Ten:  Fairmindedness 

Standard  Eleven:  Intellectual  Humility 

Standard  Twelve:  Intellectual  Courage 

Standard  Thirteen:  Intellectual  Empathy 

Standard  Fourteen:  Intellectual  Integrity 

Foundation 
Philosophy  and 
education 

Critical 

2006 

Foundation 
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Standard  Fifteen:  Intellectual  Perseverance 

Standard  Sixteen:  Confidence  in  Reason 

Standard  Seventeen:  Intellectual  Autonomy 

54.  1.  Willingness  to  Plan 

2.  Flexibility 

3.  Persistence 

4.  Willingness  to  Self-Correct,  Admit  Errors,  and  Change  Your  Mind  When  the  Evidence  Changes 

5.  Being  Mindful 

6.  Consensus  Seeking 

Halpern 

Psychology 

critical 

2003 

Halpern 

55.  Glaser’s  description  of  the  ability  to  think  critically  includes: 

an  attitude  of  being  disposed  to  consider  in  a  thoughtful  way  the  problems  and  subjects  that  come 
within  the  range  of  one's  experiences. 

Persons  who  have  acquired  a  disposition  to  want  evidence  for  beliefs,  and  who  have  acquired  an 
attitude  of  reasonableness 

Glaser 

Education 

Critical 

1941 

Glaser 

56.  Common  themes  among  some  scholars  on  dispositions: 

a  reflective  skeptical  or  questioning  attitude,  a  sensitivity  to  value-  or  ideology-laden  assumptions,  an 
insistence  on  appropriate  supporting  grounds  before  accepting  disputable  claims,  an  appreciation  of  the 
various  criteria  applicable  to  good  reasoning  and  argument  (whether  general  or  subject  dependent),  skill 
and  judgment  in  the  analysis  and  evaluation  of  claims  and  arguments,  and  a  disposition  to  be  self- 
reflective,  sensitive  to  one’s  own  possible  biases  or  assumptions. 

Reed 

Education/History 

Critical 

1998 

Ennis,  Siegel, 
Lipman, 
McPeck,  Paul, 

57.  Affective  dispositions  such  as  open-mindedness  and  diligence  in  seeking  relevant  information  are  also 
necessary  to  both  historical  thinking  and  critical  thinking 

Reed 

Education/History 

Critical 

1998 

Reed 

58.  A  critical  component  of  this  general  epistemological  orientation  is  the  disposition  to  process 

information  analytically.  Qualities  that  constitute  this  disposition,  and  the  self-report  measures  we  used 
to  roughly  index  these  qualities,  are: 

reliance  on  logic  and  distrust  of  intuition  (the  Head  Over  Heart  Scale;  HOH); 
objectivity  and  openmindedness  to  belief  revision  (the  Belief  Defensiveness  Scale;  BD); 
reflectiveness,  cognitive  vigilance,  and  the  tendency  to  monitor  one's  reasoning  (the  Need  for  Cognition 
Scale;  NFC); 

and  understanding  the  indeterminancy  of  knowledge  and  willingness  to  postpone  closure  (the  Need  for 
Closure  Scale;  NFCL) 

Klaczynski  & 
Robinson 
psychology 

Motivated 

reasoning 

2000 

Epstein, 

Pacini,  Denes- 
Raj,  Heier, 
Klaczynski, 
Fauth, 

Cacioppo, 

Petty 

Kruglanski, 

Webster, 

Klem 

59.  1.  Intellectual  curiosity,  seeking  answers  to  various  kinds  of  questions  and  problems. 

2.  Objectivity,  using  objective  factors  when  one  makes  decisions  and  avoiding  being  influenced  by 

Zechmeister  & 
Johnson 

Psychology 

Critical 

1992 

D'Angelo 
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emotional  or  subjective  factors. 

3.  Open-mindedness :  being  willing  to  consider  a  wide  variety  of  beliefs. 

4.  Flexibility,  exhibiting  a  willingness  to  change  one's  beliefs  or  methods  of  inquiry. 

5.  Intellectual  skepticism :  not  accepting  a  conclusion  as  being  true  until  adequate  evidence  is  provided. 

6.  Intellectual  honesty’’,  accepting  statements  as  true  even  when  they  don't  agree  with  one's  own  position. 

7.  Being  systematic’,  following  a  line  of  reasoning  consistently  to  a  conclusion. 

8.  Persistence’,  being  persistent  in  seeking  to  resolve  disputes. 

9.  Decisiveness’,  reaching  a  conclusion  when  the  evidence  warrants  it. 

10.  Respect  for  other  viewpoints’,  willingness  to  admit  that  you  are  wrong  and  that  others  may  be  right. 

60.  Critical  thinking  styles: 

The  Explorer 

Looks  at  all  sides  of  a  situation  or  problem 

Identifies  the  core  element  of  a  problem  or  situation 

Is  curious 

Seeks  new  developments 

The  Student 

Is  diligent 

Researches  alternative  solutions  to  problems 

Does  homework 

Corrects  errors 

The  Warrior 

Accepts  challenges 

Perseveres 

Faces  difficult  problems 

The  Navigator 

Guides  others 

Looks  ahead 

Plans  a  course  of  action 

The  Detective 

Questions  own  thoughts  and  actions 

Tolerates  ambiguity 

Pursues  the  elusive  element  of  fact 

Feldman 

Psychology 

Critical 

2002 

Feldman 

61.  Open  mindedness,  whole  heartedness,  responsibility 

Dewey 

Reflective 

1933 

Dewey 
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Education  and 
philosophy 

62.  Ideal  critical  thinker: 

The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  open-minded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  In  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  In  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  the  circumstances  of  inquiry  permit 

Facione 

Philosophy 

Critical 

1994 

APA 

63.  Inquisitiveness,  Open-mindedness,  Systematicity,  Analyticity,  Truth-seeking,  CT  Self-confidence,  and 
Maturity 

Facione 

Philosophy 

Critical 

1994 

Facione 

64.  A  well  cultivated  critical  thinker: 

•  raises  vital  questions  and  problems,  formulating  them  clearly  and  precisely; 

•  gathers  and  assesses  relevant  information,  using  abstract  ideas  to  interpret  it  effectively  comes  to 
well-reasoned  conclusions  and  solutions,  testing  them  against  relevant  criteria  and  standards; 

•  thinks  openmindedly  within  alternative  systems  of  thought,  recognizing  and  assessing,  as  need  be, 
their  assumptions,  implications,  and  practical  consequences;  and 

•  communicates  effectively  with  others  in  figuring  out  solutions  to  complex  problems 

Paul/Scriven 
Education  and 
Philosophy 

Critical 

2010 

Foundation 

65.  truth-seeking,  open-mindedness,  "analyticity,”  “systematicity,"  self-confidence,  inquisitiveness,  and 
maturity. 

Dexter,  et  al 

Nursing 

Critical 

1997 

Facione 

66.  truth-seeking,  open-mindedness,  analyticity,  systematicity,  confidence,  inquisitiveness,  and  maturity 

Walsh  & 
Seldomridge 

Nursing 

Critical 

2006 

Facione 

67.  Critical  thinkers  in  nursing  exhibit  these  habits  of  the  mind:  confidence,  contextual  perspective, 

creativity,  flexibility,  inquisitiveness,  intellectual  integrity,  intuition,  open-mindedness,  perseverance, 
and  reflection 

Staib 

Nursing 

Critical 

2003 

Scheffer  and 
Rubenfeld 

68.  Attitudes  (dispositions)of  a  critical  thinker  include  a  drive  to  formulate  a  clear  statement  of  the  thesis  or 
question  implicit  and  explicit  in  the  assigned  subject  matter;  a  willingness  to  scrutinize  and  re-examine 
attitudes,  beliefs,  and  values;  confidence  in  ability  to  think  and  willingness  to  deal  with  error  and 
mistakes;  a  sensitivity  to  identification  and  confrontation  with  problematic  situations;  a  desire  to  pursue 
new  experiences  and  a  respect  for  the  use  of  instruction  when  relevant  to  thinking  processes;  a  desire  to 
achieve  high-level  reasoning  skills  with  judgments  based  on  evidence;  a  willingness  to  scrutinize 
personal  ideas  and  values  and  to  articulate  and  empathize  with  opposing  views  without  defensive 
reactions;  a  curiosity  about  the  relationship  between  ideas,  theories,  and  systems,  as  well  as  a  passion 

Williams 

Political  Science 

Critical 

1991 

Ennis 

Ruggerio 
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for  insight  and  understanding;  and,  finally,  a  constructive  attitude  toward  argumentation  without  being 
consumed  by  ego  protection 

69.  A  critical  thinker  exhibits  the  following  dispositions  or  attitudes:  (a)  willingness  to  engage  in  and  persist 
at  a  complex  task,  (b)  habitual  use  of  plans  and  the  suppression  of  impulsive  activity,  (c)  flexibility  or 
open-mindedness,  (d)  willingness  to  abandon  nonproductive  strategies  in  an  attempt  to  self-correct,  and 
(e)  an  awareness  of  the  social  realities  that  need  to  be  overcome  (such  as  the  need  to  seek  consensus  or 
compromise)  so  that  thoughts  can  become  actions 

Halpern 

psychology 

Critical 

1998 

Flalpern 

70.  there  was  a  disposition  factor  represented  by  measures  of  need  for  cognition,  tolerance  of  ambiguity, 
and  dualistic/relativistic  thinking 

Ku 

Psychology 

Critical 

2009 

Not  stated 

71.  actively  open-minded  thinking,  openness  to  experience,  reflectivity  and  need  for  cognition 

Ku 

Psychology 

Critical 

2009 

Sa,  Clifford, 

Stanovich, 

Toplak 

72.  The  ideal  critical  thinker  is  habitually  inquisitive,  well  informed,  trustful  of  reason,  open-minded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  pmdent  in  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  the  circumstances  of  inquiry  permit. 

Facione 

Philosophy 

Critical 

1995 

APA 

73.  Inquisitiveness,  Open-mindedness,  Systematicity,  Analyticity,  Truth-seeking,  CT  Self-confidence,  and 
Maturity) 

Facione 

Philosophy 

Critical 

1995 

Facione 

74.  It  is  hard  to  go  very  far  into  the  core  concept  of  the  critical  person,  however,  without  recognizing  the 
centrality  of  multilogical  thinking,  the  ability  to  think  accurately  and  fainnindedly  within  opposing 
points  of  view  and  contradictory  frames  of  reference. 

Paul 

Philosophy  & 
Education 

Critical 

Rational 

1987 

Paul 

75.  If  the  student  has  developed  genuine  open-mindedness,  Ennis  claims,  he  will:  (a)  Consider  seriously 
from  premises  with  which  one  disagrees — without  letting  the  disagreement  interfere  with  one's 
reasoning  ("suppositional  thinking");  (c)  withhold  judgment  when  the  evidence  and  reasons  are 
insufficient 

Paul 

Philosophy  & 
education 

Critical 

Rational 

1987 

Ennis 

76.  To  think  critically  in  this  sense  requires,  as  Passmore  points  out,  initiative,  independence,  courage, 

(and)  imagination. 

Paul 

Philosophy  & 
education 

Critical 

Rational 

1987 

Passmore 

77.  It  also  describes  one's  inclination  to  use  one's  critical  thinking  skills  as  manifested  in  the  habits  of  mind 
known  as  inquisitiveness,  open-mindedness,  truth-seeking,  analyticity,  systematicity,  reasoning  self- 
confidence,  and  cognitive  maturity 

Facione 

Nursing 

Critical 

1997 

Facione 

78.  Motivations: 

Molden  &  Fliggins 

Good 

2005 

Various 

103 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-20 14-2455;  Cleared  21  May  2014 


Dispositions 

Author/Field 

Type  of 
Thinking 

Date 

Source 

Outcome  motivated  thinking 

Directional  includes  positive  self  evaluation  and  positive  eval  of  others 

Non-directional  includes  desires  for  accuracy  desires  for  clarity  and  conciseness,  or  closure 

Also  mentioned  need  for  cognition  in  the  notes  of  the  article 

Strategy  motivated  thinking 

Promotion  focus  -  promoting  positive  outcome 

Prevention  focus  -  avoidance  of  negative  outcome 

Psychology 

79.  openmindedness,  caution,  and  valuing  being  well-informed,  the  dispositions  to  seek  and  be  sensitive  to 
alternatives,  and  to  be  cautious  but  not  totally  skeptical 

Ennis  &  Millman 
Philosophy  and 
education 

Critical 

2005 

Ennis 

80.  A  student  who  has  developed  critical  thinking  dispositions  will  show  the  inclination  to 

1 .  approach  questions  with  an  open-minded  and  curious  attitude,  be  informed  by  multiple  relevant 
perspectives,  and  be  willing  to  examine  questions  in  a  fair-minded  way; 

2.  apply  critical  thinking  skills  to  thinking  about  issues  in  a  major  field  of  study,  general  education 
courses,  and  everyday  lives; 

3.  reflect  on  how  best  to  answer  questions,  solve  problems,  and  make  decisions  in  academic  and 
everyday  settings,  (p.3) 

Bensley  et  al 
Frostburg  State 
University 

Education 

Critical 

2006 

Frostburg 

81.  The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  open-minded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  in  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  the  circumstances  of  inquiry  permit,  (pp.  3-4) 

Spicer  &  Flanks 
Education 

Critical 

1995 

APA 

82.  1.  inquisitiveness;  2.  open  mindedness;  3.  systematicity;  4.  analyticity;  5.  truth-seeking;  6.  CT 
selfconfidence;  and  7.  maturity. 

Spicer  &  Flanks 
Education 

Critical 

1995 

Facione 

83.  A  critical  thinker  will  exhibit  the  following  dispositions  or  attitudes: 

Willingness  to  engage  in  and  persist  at  a  complex  task 

Willingness  to  plan 

Flexibility  or  open-mindedness 

Willingness  to  self  correct 

Being  mindful 

Consensus-Seeking  (pp.  35-36) 

Greenwood 

Education 

Critical 

1992 

Halpern 

84.  1 .  Be  curious  and  inquire  about  how  and  why  things  work. 

2.  Be  organized,  orderly,  and  focused  in  inquiry  or  in  thinking. 

3.  Willingly  persevere  and  persist  at  a  complex  task. 

NPEC 

Education 

Critical 

2000 

Jones  et  al 
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4.  Be  flexible  and  creative  in  seeking  solutions. 

5.  Be  inclined  to  arrive  at  a  reasonable  decision  in  situations  where  there  is  more  than  one  plausible 
solution. 

6.  Apply  insights  from  cultures  other  than  their  own. 

7.  Exhibit  honesty  in  facing  up  to  their  prejudices,  biases,  or  tendency  to  consider  a  problem  solely  from 
their  viewpoint. 

8.  Monitor  their  understanding  of  a  situation  and  progress  toward  goals. 

9.  Find  ways  to  collaborate  with  others  to  reach  consensus  on  a  problem  or  issues. 

10.  Be  intellectually  careful  and  precise. 

1 1 .  Value  the  application  of  reason  and  the  use  of  evidence. 

12.  Be  open-minded;  strive  to  understand  and  consider  divergent  points  of  view. 

13.  Be  fair-minded;  seek  truth  and  be  impartial,  even  if  the  findings  of  an  inquiry  may  not  support  one’s 
preconceived  opinions. 

14.  Willingly  self-correct  and  learn  from  errors  made  no  matter  who  calls  them  to  our  attention,  (pp.  23- 
25) 

85.  be  curious  and  inquire  about  how  and  why  things  work 
flexibility  and  creativity  in  seeking  solutions 

exhibit  honesty  in  facing  up  to  one's  prejudices,  biases,  or  tendencies  to  consider  a  problem  solely  from 
one's  own  viewpoint. 

willingness  to  persevere  and  persist  at  a  complex  task, 
being  organized,  orderly,  and  focused  in  inquiry  or  thinking 

the  inclination  to  arrive  at  a  reasonable  decision  in  situations  where  there  is  more  than  one  plausible 
solution. 

find  ways  to  collaborate  with  others  to  reach  a  consensus  on  a  problem  or  issue 

monitor  one's  understanding  of  a  situation  and  progress  towards  goals 

the  inclination  to  apply  insights  from  cultures  other  than  one's  own  was  rated  important 

willingly  self-correct  and  learn  from  errors  made  no  matter  who  calls  them  to  their  attention 

be  fair-minded 

seek  truth  and  be  impartial,  even  if  findings  may  not  support  one's  preconceived  opinions 
be  open-minded  and  strive  to  understand  and  consider  different  points  of  view 
value  the  application  of  reason  and  the  use  of  evidence 
be  intellectually  careful  and  precise 

Jones  et  al 

Education 

Critical 

1995 

Delphi  process 
with  faculty 
and  employers 

86.  Affective  dispositions  to  critical  thinking  were  divided  into  two  categories  (1)  approaches  to  life  and 

Pierce 

Critical 

2006 

APA 
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living  in  general,  and  include  such  traits  as  inquisitiveness,  trust  in  reason,  and  fairmindedness,  and  (2) 
approaches  to  specific  issues,  questions  or  problems,  and  include  such  traits  as  clarity  in  stating  the 
question,  diligence  in  seeking  relevant  information,  and  persistence  although  difficulties  are 
encountered. 

Education 

87.  The  disposition  to  be  broad  and  adventurous. 

The  disposition  toward  wondering,  problemfinding,  and  investigating. 

The  disposition  to  build  explanations  and  understandings. 

The  disposition  to  make  plans  and  be  strategic. 

The  disposition  to  be  intellectually  accurate. 

The  disposition  to  ask  for  and  evaluate  reasons. 

The  disposition  to  be  metacognitive. 

Tishman 

Education 

Good 

1994 

Tishman 

88.  •  . .  .careful  attention  to  what  you  know,  vigilance  for  what  you  do  not,  and  the  courage  to  question  both 
of  the  above  categories. 

•  . .  .thinking  about  an  topic,  issue,  or  challenge  in  a  way  that  sets  aside  my  immediate  ’gut’  response, so 
that  I  can  be  open  and  reflective  to  other  possible  ways  of  viewing  the  challenge. . . 

•  . .  .thinking  deeply,  keenly,  flexibly,  openly,  reflectively,  with  an  awareness  of  self  and  others,  with 
attention  to  what  is  known  and  unknown,  and  with  humility. 

Krupat 

Medicine 

Critical 

2008 

Survey  of 
physicians 

89.  critical  thinking  has  two  important  dimensions:  a  "frame  of  mind"  and  a  number  of  specific  mental 

operations.  The  frame  of  mind  includes  an  alertness  to  the  need  to  evaluate  information,  a  willingness  to 
test  opinions,  and  a  desire  to  consider  all  viewpoints. 

Taube 

Education 

Critical 

1995 

Beyer 

90.  a  critical  "attitude"  consisting  of  doubt,  carefulness,  objectivity,  and  determinism,  as  well  as  thinking 
strategies  and  thinking  skills,  in  their  model  of  critical  thinking 

Taube 

Education 

Critical 

1995 

DeNitto  and 
Strickland 

91.  Paul  and  Nosich  stated  that  "critical  thinking  entails  the  possession  and  active  use  of  a  set  of  traits  of 
mind"  including  "independence  of  thought,  fairmindedness,  intellectual  humility,  intellectual  courage, 
intellectual  perseverance,  intellectual  integrity,  curiosity,  confidence  in  reason,  the  willingness  to  see 
objections,  to  enter  sympathetically  into  another's  point  of  view,"  and  "to  recognize  one's  own 
egocentricity  or  ethnocentricity 

Taube 

Education 

Critical 

1995 

Paul  &  Nosich 

92.  The  ideal  critical  thinker  is  habitually  inquisitive,  well-informed,  trustful  of  reason,  openminded, 
flexible,  fair-minded  in  evaluation,  honest  in  facing  personal  biases,  prudent  in  making  judgments, 
willing  to  reconsider,  clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and  persistent  in  seeking  results 
which  are  as  precise  as  the  subject  and  circumstances  of  inquiry  permit 

Taube 

Education 

Critical 

1995 

Facione/APA 

93.  According  to  Ennis  (1991,  Table  1),  the  dispositions  of  the  ideal  critical  thinker  are  as  follows: 

Taube 

Critical 

1995 

Ennis 
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(1)  To  be  clear  about  the  intended  meaning  of  what  is  said,  written,  or  otherwise  communicated. 

(2)  To  determine  and  maintain  focus  on  the  conclusion  or  question. 

(3)  To  take  into  account  the  total  situation. 

(4)  To  seek  and  offer  reasons. 

(5)  To  try  to  be  well  informed. 

(6)  To  look  for  alternatives. 

(7)  To  seek  as  much  precision  as  the  situation  requires. 

(8)  To  try  to  be  reflectively  aware  of  one's  own  basic  beliefs. 

(9)  To  be  open-minded,  consider  seriously  viewpoints  other  than  one’s  own 

(10)  To  withhold  judgment  when  the  evidence  and  reasons  are  insufficient. 

(1 1)  To  take  a  position  (and  change  a  position)  when  the  evidence  and  reasons  are  sufficient  to  do  so. 

(12)  To  use  one's  critical  thinking  abilities, 

Education 

94.  Siegel  identified  two  components  essential  to  critical  thinking:  reason  assessment,  which  parallels  the 
thinking  abilities  outlined  so  far,  and  the  "critical  spirit,"  which  includes  attitudes,  dispositions,  habits  of 
mind,  and  character  traits,  all  of  which  combine  to  produce  a  "be  of  reason"  characterized  by 
intellectual  honesty,  objectivity,  justice  to  evidence,  and  sympathetic  and  impartial  consideration  of 
interests 

Taube 

Education 

Critical 

1995 

Siegel 

95.  The  situation  of  dispositions  and  traits  being  an  important  and  largely  untested  component  of  mental 
skills  has  been  known  for  many  years.  Wechsler  (1943,  1950)  identified  "nonintellective"  factors  as 
untested  content  in  intelligence  tests.  These  "non-intellective"  factors  include  drive,  energy,  curiosity, 
anxiety,  impulsiveness,  persistence,  and  other  temperamental  factors 

Taube 

Education 

Critical 

1995 

Wechsler 

96.  It  seems  plausible  that  such  personality  traits  as  open-mindedness,  cognitive  complexity,  need  for 
cognition,  tolerance  of  ambiguity,  and  reflectiveness  represent  the  dispositional  component  of  critical 
thinking,  and  are  characteristic  of  effective  critical  thinkers 

Taube 

Education 

Critical 

1995 

Taube 

97.  Truth-Seeking,  or  the  disposition  to  seek  the  truth,  ask  questions,  and  be  honest  and  objective  about 
pursuing  inquiry  even  if  the  findings  contradict  one's  interests  or  opinions;  Open-Mindedness,  or  the 
tendency  to  be  tolerant  of  divergent  views  and  sensitive  to  one's  own  bias;  Analyticity,  defined  as  being 
alert  to  potentially  problematic  situations,  anticipating  possible  results  and  consequences,  and  prizing 
the  use  of  reason  or  evidence;  Systematicity,  or  the  tendency  to  be  organized,  orderly,  focused,  and 
diligent  in  inquiry;  Self-Confidence,  or  trust  in  one's  own  reasoning  processes;  Inquisitiveness,  or 
intellectual  curiosity;  and  Maturity,  or  the  disposition  to  make  reflective  judgments. 

Taube 

Education 

Critical 

1995 

Facione 

98.  Task  orientation  and  ego  orientation 

Taube 

Education 

Critical 

1995 

Nicholls 
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99.  Critical  thinkers  are  disposed  to  seek  reasons,  try  to  be  well  informed,  use  credible  sources  and  mention 
them,  look  for  alternatives,  consider  seriously  points  of  view  other  than  their  own,  withhold  judgment 
when  the  evidence  and  reasons  are  insufficient,  seek  as  much  precision  as  the  subject  permits,  among 
other  activities 

Norris 

Education 

Critical 

1989 

Ennis  &  Norris 

100.  Seek  a  statement  of  the  thesis  or  question 

Seek  reasons 

Try  to  be  well  informed 

Use  credible  sources  and  mention  them 

Take  into  account  the  total  situation 

Keep  their  thinking  relevant  to  the  main  point 

Keep  in  mind  the  original  or  most  basic  concern 

Look  for  alternatives 

Are  open  minded  and  seriously  consider  points  of  view  other  than  their  own;  reason  from  starting  points 
with  which  they  disagree  without  letting  disagreement  interfere  with  their  reasoning 

Take  a  position  and  change  a  position  when  evidence  and  reasons  are  sufficient  to  do  so 

Seek  as  much  precision  as  the  subject  permits 

Deal  in  an  orderly  manner  with  parts  of  a  complex  whole 

Employ  their  critical  thinking  abilities 

Are  sensitive  to  the  feelings,  level  of  knowledge,  and  degree  of  sophistication  of  others 

McKown 

Military 

Critical 

1997 

Ennis  &  Norris 

101. (a)  Open-mindedness,  (b)  Analycity,  (c)Truth-seeking,  (d)  Systematicity,  (e)  Self-confidence, 
(f)lnquisitiveness,  and  (g)  Maturity 

Valanides  and 

Angeli 

Education 

Critical 

2008 

Facione 

102. Disposition  to  think  critically  involves,  among  other  traits,  such  factors  as  the  inclination  to  ask 

challenging  questions  and  follow  the  reasons  and  evidence  wherever  they  lead,  tolerance  for  new  ideas, 
willingness  to  use  reason  and  evidence  to  solve  problems,  and  willingness  to  see  complexity  in 
problems 

Pascarella  and 
Terenzini 

Education 

Critical 

2005 

Facione 

103. disposition  to  weigh  new  evidence  that  conflicts  with  a  held  belief,  to  spend  a  considerable  amount  of 
time  on  a  problem  before  giving  up,  and  to  evaluate  others'  opinions  in  order  to  form  one's  own.  They 
are  dispositions  toward  open-minded  thinking  and  have  strong  similarities  with  two  epistemological 
dimensions  about  the  nature  of  knowledge  as  defined  by  Schommer 

Mason 

Psychology 

epistemolo 

gy 

2003 

Baron 

104. Inspired  by  the  above  definitions,  we  may  tentatively  suggest  that  a  critical  thinker  has  the  following 
sorts  of  dispositions,  supported  by  relevant  abilities  and  attitudes: 

1.  to  examine  and  evaluate  claims  to  knowledge,  nonnative  principles,  theories,  policy 

Hare 

Education 

critical 

1999 

Hare 

108 

Distribution  A.  Approved  for  public  release;  distribution  unlimited. 
88ABW-20 14-2455;  Cleared  21  May  2014 
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Author/Field 

Type  of 
Thinking 

Date 

Source 

recommendations,  and  other  matters  where  judgement  is  called  for; 

2.  to  determine  if  what  is  presented  as  evidence  and  relevant  argument  merits  being  so  considered,  and 
to  what  extent  it  is  biased,  inadequate,  misleading  or  in  other  ways  deficient; 

3.  to  resist  efforts  by  others  to  impose  ideas  on  him  or  her,  and  to  avoid  being  imposed  upon  by  ideas 
which  are  taken  for  granted  in  the  prevailing  intellectual  and  social  climate; 

4.  to  regard  situations  and  issues  conventionally  deemed  to  be  straightforward  as  potentially 
problematic  and  controversial,  to  try  to  attend  to  the  unusual  when  attracted  by  the  familiar,  and  to 
imagine  ways  in  which  the  existing  framework  might  be  transcended. 

105.  actively  open-minded  thinking,  need  for  cognition  (the  tendency  to  think  a  lot),  consideration  of  future 
consequences,  typical  intellectual  engagement,  need  for  closure,  superstitious  thinking,  and  dogmatism 

Stanovich 

Psychology 

Critical 

Rational 

2011 

Not  stated 

106.  individual  differences  on  our  index  of  argument-driven  processing  can  be  predicted  by  measures  of 
dogmatism  and  absolutism,  categorical  thinking,  openness  ,  flexible  thinking,  belief  identification, 
counterfactual  thinking,  superstitious  thinking,  and  actively  open-minded  thinking. 

Stanovich 

Psychology 

Critical 

Rational 

2011 

Various 

107.  care  that  their  beliefs  be  true,  and  that  their  decisions  be  justified;  care  to  present  a  position  honestly 
and  clearly;  and  care  about  the  worth  of  other  people’s  opinions 

Ortiz 

Philosophy 

Critical 

2007 

Ennis 

108. Examples  might  be  the  disposition  to  weigh  new  evidence  against  a  favored  belief  heavily  (or  lightly), 
the  disposition  to  spend  a  great  deal  of  time  (or  very  little)  on  a  problem  before  giving  up,  or  the 
disposition  to  weigh  heavily  the  opinions  of  others  in  forming  one's  own.  Although  you  cannot  improve 
working  memory  by  instruction,  you  can  tell  someone  to  spend  more  time  on  problems  before  she  gives 
up,  and  if  she  is  so  inclined,  she  can  do  what  you  say. 

Baron 

Psychology 

Rational 

Intelligent 

thinking 

1985 

Baron 

109.  Need  for  closure 

Kruglanski 

psychology 

Decision 

making 

2003 

Kruglanski 

1 10.  Need  for  cognition 

Cacioppo,  Petty, 
Feinstein,  Jarvis 
Psychology 

General 

thinking 

1996 

Cohen 

1 11. Information  processing  stvles,  based  on  Epstein’s  cognitive  experimental  self  theorv.  are: 

Intuitive 

Rational 

Klaczynski, 

Gordon,  Klauth 
Psychology 

Critical 

1997 

Epstein 

1 12. Nine  distinct  but  related  personality  constructs  were  defined,  as  follows: 

(a)  Typical  intellectual  engagement  is  the  core  construct  of  the  hypothesized  construct  space.  Scale 
items  were  expected  to  differentiate  among  individuals  in  their  typical  expression  of  a  desire  to  engage 
and  understand  their  world,  their  interest  in  a  wide  variety  of  things,  and  their  preference  for  a  complete 
understanding  of  a  complex  topic  or  problem,  a  need  to  know; 

Goff,  Ackerman 
Psychology 

General 

1992 

Goff, 

Ackerman 

109 
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(b)  hard  work  items  indicate  a  preference  for  tasks  or  goals  that  are  difficult  to  attain  or  require 
persistence; 

(c)  perfectionism  describes  persons  who  tend  to  value  high  accuracy  or  excellence; 

(d)  openness  involves  preference  for,  or  high  tolerance  of,  variety  or  new  experiences; 

(e)  absorption  describes  situations  in  which  people  find  themselves  so  deeply  involved  in  an  intellectual 
activity  that  cognizance  of  time  and  the  immediate  environment  is  decreased  markedly; 

(f)  distractibility  may  discourage  intellectual  engagement  by  preventing  initial  or  sustained  focus  on  a 
task  or  problem; 

(g)  extroverted  intellectual  engagement  involves  enjoyment  of  engagement  with  others  in  intellectual 
activities; 

(h)  introverted  intellectual  engagement  occurs  when  a  person  tends  to  spend  much  time  analyzing  their 
own  thinking  or  wondering  about  the  thinking  of  others; 

(i)  energy  is  expected  to  represent  a  person's  general  energy  level;  and 

(j)  four  interest  areas  were  included,  namely,  interest  in  arts  and  humanities,  science,  social  science,  and 
technology 

1 13. Need  for  cognition 

Verplanken 

psychology 

General 

1993 

Cohen  and 
Cacioppo, 

Petty 

114.  16  habits  of  mind  are: 

1 .  Persisting 

2.  Managing  Impulsivity 

3.  Listening  To  Others — With  Understanding  and  Empathy 

4.  Thinking  Flexibly 

5.  Thinking  About  our  Thinking  (Metacognition) 

6.  Striving  For  Accuracy  and  Precision 

7.  Questioning  and  Posing  Problems 

8.  Applying  Past  Knowledge  to  New  Situations 

9.  Thinking  and  Communicating  with  Clarity  and  Precision 

10.  Gathering  Data  through  All  Senses 

1 1 .  Creating,  Imagining,  and  Innovating 

12.  Responding  with  Wonderment  and  Awe 

13.  Taking  Responsible  Risks. 

14.  Finding  Humor 

Costa  and  Kallick 
Education 

Effective 

2000 

Costa  and 

Kallick; 

various 

sources 

110 
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15.  Thinking  Interdependently 

16  Learning  Continuously 

1 15. Need  for  closure 

Kruglanski  & 
Webster 

General 

1994 

Kruglanski  & 
Webster 

1 16. Two  primary  dispositions  described: 

Impulsivity/reflection 

Fairness/bias 

Baron 

psychology 

General 

1985 

various 

1 17. Government  metaphor  for  thinking  styles: 

Functions 

Legislative 

Executive 

Judicial 

Forms 

Monarchic 

Hierarchic 

Oligarchic 

Anarchic 

Levels 

Global 

Local 

Scope 

Internal 

External 

Leanings 

Liberal 

Conservative 

Sternberg 

Psychology 

General 

1997 

Sternberg 

1 18. Field  Dependence-Independence 

Sternberg 

Psychology 

General 

1997 

Witkin 

1 19.Impulsivity-Reflectivity 

Sternberg 

Psychology 

General 

1997 

Kagan 

120. Need  for  cognition 

Kardash  and 

Scholes 

Psychology 

General 

1996 

Cacioppo, 
Petty,  &  Kao 

121. Government  metaphor  for  thinking  styles: 

Functions 

Legislative 

Sternberg 
Grigorenko,  and 
Zhang 

Psychology  and 

Successful 

intelligence 

2008 

Sternberg 

Ill 
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Executive 

Judicial 

Forms 

Monarchic 

Hierarchic 

Oligarchic 

Anarchic 

Levels 

Global 

Local 

Scope 

Internal 

External 

Leanings 

Liberal 

Conservative 

education 

122.  analytical,  inquisitive,  judicious,  open-minded,  systematic,  truth-seeking  and  possess  cognitive 
maturity 

Willis 

Industrial 

psychology 

Critical 

2004 

Facione 

123. Need  for  cognition 

Cacioppo,  Petty, 

Kao 

Psychology 

General 

1984 

Cacioppo, 

Petty,  Kao 

124.Truthseeking,  open-mindedness;  analyticity,  systematicity,  critical  thinking  self  confidence, 
inquisitiveness,  maturity  of  judgment 

Facione  &  Facione 
Philosophy 

Critical 

2010 

Facione 

125. cognitive  reflection 

Toplak,  West, 

Stanovich 

Psychology 

Rational 

2011 

Frederick 

126. need  for  cognition,  stimulation  seeking,  critical  thinking,  open-mindedness 

Ormrod 

Education 

learning 

2011 

various 

127.  Rational  and  ethical  qualities  of  critical  thinking 

Clarity 

Depth 

Precision 

Completeness 

Specificity 

Fairness 

Haase 

Critical 

2010 

Haase 

112 
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Relevance 

Adequacy 

Accuracy 

Evidence 

Consistency 

Logicalness 

Significance 

128.  Ennis  list: 

1 .  Seek  a  clear  statement  of  the  thesis  or  question. 

2.  Seek  reasons. 

3.  Try  to  be  well  informed. 

4.  Use  credible  sources  and  mention  them. 

5.  Take  into  account  the  total  situation. 

6.  Try  to  remain  relevant  to  the  main  point. 

7.  Keep  in  mind  the  original  and/or  basic  concern. 

8.  Look  for  alternatives. 

9.  Be  open-minded. 

a.  Consider  seriously  points  of  view  other  than  one's  own  ("dialogical  thinking"). 

b.  Reason  from  premises  with  which  one  disagrees — without  letting  the  disagreement  interfere  with 
one's  reasoning  ("suppositional  thinking"). 

c.  Withhold  judgment  when  the  evidence  and  reasons  are  insufficient. 

10.  Take  a  position  (and  change  a  position)  when  the  evidence  and  reasons  are  sufficient  to  do  so. 

11.  Seek  as  much  precision  as  the  subject  permits. 

12.  Deal  in  an  orderly  manner  with  the  parts  of  a  complex  whole. 

13.  Be  sensitive  to  the  feelings,  level  of  knowledge,  and  degree  of  sophistication  of  others. 

Swartz 

Education 

Critical 

1987 

Ennis 

129.  truth-seeking,  open-mindedness,  analyticity,  systematicity,  critical  thinking  self  confidence, 
inquisitiveness,  and  maturity  of  judgment 

McCarthy 

Military 

Critical 

2001 

Facione 

130.  truth-seeking,  open-mindedness,  analyticity,  systematicity,  self  confidence,  inquisitiveness 

Cubukcu 

Education 

Critical 

2006 

Facione 

131. truth-seeking,  open-mindedness,  analyticity,  systematicity,  critical  thinking  self  confidence, 
inquisitiveness,  and  maturity  of  judgment 

Begbie 

Nursing 

Critical 

2007 

Facione 

132.  truth-seeking,  open-mindedness,  analyticity,  systematicity,  critical  thinking  self  confidence, 
inquisitiveness,  and  maturity  of  judgment 

Lederer 

Medicine 

Critical 

2007 

Facione 

113 
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133. Synthesis  of  various  lists  of  thinking  dispositions: 

1.  Creative  thinking:  looking  out,  up,  around,  and  about 

Open-minded 

Curious 

2.  Reflective  thinking:  looking  within 

Metacognitive 

3.  Critical  thinking:  looking  at,  through,  and  in  between 

Seeking  truth  and  understanding 

Strategic 

Skeptical 

Ritchart 

Education 

Critical 

Reflective 

Creative 

2002 

various 

134. Effective  problem  solvers,  in  contrast,  view  problems  as  challenges  and  are  persistent  in  seeking 
solutions,  if  a  particular  strategy  is  unsuccessful,  they  take  a  different  approach.  Other  attitudes  and 
habits  of  effective  thinking  are  well  known:  open-mindedness,  flexibility,  ability  to  defer  immediate 
judgment,  attention  to  detail. 

McTighe 

Education 

Effective 

1987 

McTighe 

135.  Provided  a  more  concise  summary  of  her  earlier  seven  dispositions: 

1.  The  disposition  to  pose  and  explore  problems. 

2.  The  disposition  to  critique  and  test  theories  and  explanations. 

3.  The  disposition  to  seek  multiple  perspectives  and  possibilities. 

4.  The  disposition  to  be  judicious  and  reflective. 

Tishman 

Education 

Good 

2001 

Tishman 

136.  Siegel  (1988)  does  also,  as  well  as  having  a  critical  spirit,  for  which  he  offers  several  subdispositions: 
objectivity,  intellectual  honesty,  impartiality,  a  willingness  to  conform  j  udgments  and  actions  to 
principle,  and  a  commitment  to  seek  and  evaluate  reasons. 

Ennis 

Philosophy 

critical 

1996 

Siegel 

137.Facione,  Sanchez,  &  Facione,  1994)  offer  seven  disposition  factors: 

openmindedness,  inquisitiveness,  systematicity,  analyticity,  truth-seeking,  critical  thinking  self- 
confidence,  and  maturity. 

Ennis 

Philosophy 

critical 

1996 

Facione 

138. 1.  To  be  broad  and  adventurous 

2.  Toward  sustained  intellectual  activity 

3.  To  clarify  and  seek  understanding 

4.  To  be  planful  and  strategic 

5.  To  be  intellectually  careful 

6.  To  seek  and  evaluate  reasons 

7.  To  be  metacognitive 

Ennis 

Philosophy 

critical 

1996 

Tishman 

139. 1.  Care  that  their  beliefs  be  true,  and  that  their  decisions  be  justified;  that  is,  care  to  "get  it  right"  to  the 
extent  possible,  or  at  least  care  to  do  the  best  they  can.  This  includes  the  interrelated  dispositions  to  do 

Ennis 

Philosophy 

critical 

1996 

Ennis 

114 
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the  following: 

A.  Seek  alternatives  (hypotheses,  explanations,  conclusions,  plans,  sources),  and  be  open 
to  them; 

B.  Endorse  a  position  to  the  extent  that,  but  only  to  the  extent  that,  it  is  justified  by  the 
information  that  is  available; 

C.  Be  well-informed;  and 

D.  Seriously  consider  points  of  view  other  than  their  own. 

2.  Represent  a  position  honestly  and  clearly  (theirs  as  well  as  others').  This  includes  the 
dispositions  to  do  the  following: 

A.  Be  clear  about  the  intended  meaning  of  what  is  said,  written,  or  otherwise 
communicated,  seeking  as  much  precision  as  the  situation  requires; 

B.  Determine,  and  maintainfocus  on,  the  conclusion  or  question; 

C.  Seek  and  offer  reasons; 

D.  Take  into  account  the  total  situation;  and 

E.  Be  reflectively  aware  of  their  own  basic  beliefs. 

3.  Care  about  the  dignity  and  worth  of  every  person.  This  includes  the  dispositions  to: 

A.  Discover  and  listen  to  others'  view  and  reasons; 

B.  Take  into  account  others  feelings  and  level  of  understanding,  avoiding  intimidating  or 
confus  ing  others  with  their  critical  thinking  prowess;  and 

C.  Be  concerned  about  others'  welfare. 

140.  1.  The  disposition  to  pose  and  explore  problems. 

2.  The  disposition  to  critique  and  test  theories  and  explanations. 

3.  The  disposition  to  seek  multiple  perspectives  and  possibilities. 

4.  The  disposition  to  be  judicious  and  reflective. 

Tishman 

Education 

Good 

2001 

Perkins,  Jay, 
Tishman 

115 
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APPENDIX  B  -  Thinking  Disposition  Related  Competencies  Identified  for  Intelligence  Analysts 


This  table  contains  disposition-related  competencies  for  intelligence  analysts  found  during  the  search.  Source  indicates  the  original  source  of  the  list. 


Analyst  Characteristics 

Author 

Date 

Source 

1.  Characteristics  required  for  Descriptive  intelligence: 

Self-Motivated 

Observant 

Intensely  Concentrates 

Moore 

2001 

Moore  & 

Krizan 

2.  Characteristics  required  for  Explanatory  intelligence: 

Self-Motivated 

Observant 

Intensely  Concentrates 

Questions  Convention 

Moore 

2001 

Moore  & 

Krizan 

3.  Characteristics  required  for  interpretive  intelligence: 

Exhibits  a  Sense  of  Wonder 

Self-Motivated 

Fascinated  with  Puzzles 

Exhibits  A-ha  Thinking 

Observant 

Reads 

Fruitfully  Obsessed 

Intensely  Concentrates 

Takes  Variable  Perspectives 

Makes  Creative  Connections 

Questions  Convention 

Exhibits  a  Sense  of  Humor 

Playful 

Moore 

2001 

Moore  & 

Krizan 

4.  Characteristics  required  for  Estimative  intelligence: 

Curious 

Exhibits  a  Sense  of  Wonder 

Self-Motivated 

Fascinated  with  Puzzles 

Exhibits  A  -Ha  Thinking 

Observant 

Moore 

2001 

Moore  & 

Krizan 
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Date 
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Reads 

Fruitfully  Obsessed 
intensely  concentrates 

Takes  Variable  Perspectives 

Makes  Creative  Connections 

Questions  Convention 

Exhibits  a  Sense  of  Humor 

Playful 

5.  O’Connor  provided  several  paragraphs  listing  characteristics: 

Intelligence  analysts  must  know  themselves.  They  must  understand  their  own  lenses  that  they  use  to  process, 
filter,  channel,  or  focus  information.  These  lenses  are  known  by  many  names  -  mental  models,  mind-sets,  or 
analytical  assumptions. 

Intelligent  analysts  tend  to  treat  what  they  expect  to  perceive  as  more  important  than  what  they  want  to 
perceive.  They  don’t  engage  in  wishful  thinking;  they  think  reflectively.  They  have  analyzed  their  own 
background,  not  in  a  soul-searching  sense  of  trying  to  find  what  they  really  want,  but  in  terms  of  looking  at 
how  past  experiences,  education  and  training,  cultural  and  organizational  norms  have  influenced  them  to  pay 
particular  attention  to  some  things  and  not  to  others. 

Intelligence  analysis  doesn’t  involve  an  open  mind.  There’s  no  such  thing  as  an  open  mind,  no  matter  how  you 
define  it.  Preconceptions  are  inevitable.  Intelligence  analysts  obtain  objectivity  by  making  basic  assumptions 
and  reasoning  as  explicitly  as  possible. 

Analysts  commonly  try  to  shift  back  and  forth  from  one  perspective  to  another.  They  try  to  perceive  things 
from  an  adversary’s  interpretation  as  well  as  from  the  United  State’s  point  of  view. 

O'Connor 

2004 

Not  stated 

6.  The  attributes  of  the  ideal  analyst  offered  by  analysts  Interviewed  during  the  course  of  this  project  are 
summarized  as  follows.  The  ideal  analyst: 

•  is  a  technologist 

•  Is  focused  (either  a  specialist  or  generalist,  not  both]. 

•  Is  an  Information  entrepreneur as  described  earlier). 

•  Is  comfortable  with  changing  roles  as  apprentice,  peer,  trainer,  or  consultant. 

•  Can  communicate  (written  and  oral). 

•  Is  a  detective 

•  Is  imaginative 

•  Is  self-starting,  self  organizing 

•  Has  a  profession  (Intelligence  analysis). 

•  Has  related  hobbies  or  technology  Interests. 

Katter, 

Montgomery, 

Thompson 

1979 

Surveyed 

analysts 
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Date 

Source 

• 

Prefers  analysis  to  management. 

• 

Can  perform  multiple,  concurrent  activities. 

• 

Is  self-confident. 

• 

Has  a  photographic  memory. 

7. 

Moore  &  Krizan  proposed: 

Moore  &  Krizan 

2005 

Moore  & 

Insatiably  curious 

Self  Motivated 

Fascinated  by  puzzles 

Exhibits  AHA  thinking 

Observes  voraciously 

Reads  voraciously 

Fruitfully  obsessed 

Takes  variable  perspectives 

Makes  creative  connections 

Playful 

Has  a  sense  of  humor 

Has  sense  of  wonder 

Concentrates  intensely 

Questions  conventions 

Krizan 

8. 

A  significant  percentage  (21  percent)  of  those  who  choose  to  pursue  employment  in  national  security 
intelligence  tend  to  express  the  following  behavior  preferences: 

orientation  to  the  inner  world  of  ideas  rather  than  the  outer  world  of  things  and  people,  tendency  to  gather 
factual  information  through  the  senses  rather  than  inspiration,  proclivity  to  make  decisions  on  the  basis  of 
logic  rather  than  emotion,  and  an  eagerness  to  seek  closure  proactively  instead  of  leaving  possibilities  open. 

Krizan 

1999 

MBTI 

9. 

Insatiably  curious 

Moore  &  Krizan 

2001 

Moore  & 

Self  Motivated 

Krizan 

Fascinated  by  puzzles 

Exhibits  AHA  thinking 

Observes  voraciously 

Reads  voraciously 

Fruitfully  obsessed 

Takes  variable  perspectives 

Makes  creative  connections 

Playful 

Has  a  sense  of  humor 

Has  sense  of  wonder 

Concentrates  intensely 
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Questions  conventions 

10.  But,  a  good  intelligence  analyst  is  more  than  just  an  analyst.  He  or  she  should  also  have  a  set  of  fairly  unique 
personal  attributes.  Among  these  are: 

The  ability  to  deal  with  a  high  level  of  ambiguity  and  lack  of  structure. 

The  ability  to  be  a  self-starter. 

The  ability  to  be  comfortable  and  articulate  with  people  in  the  organization  who  are  considerably  more  senior 
then  he  or  she  is. 

The  ability  to  get  the  customer  to  define  what  is  really  needed  and  why. 

The  humility  to  understand  that,  as  a  support  person,  an  intelligence  analyst  is  not  an  advocate. 

The  personal  fortitude  to  push  an  intelligence  finding  so  that  it  gets  the  attention  it  deserves  by  senior 
managers  —  even  if  this  means  crossing  the  formal  hierarchy  of  the  company. 

Sullivan 

2002 

Sullivan 

11.  Characteristics 

•  Curious 

•  Creative 

•  Conscientious 

•  Passionate 

Unknown 
(RW  615) 

Undated 

Unknown 

12.  Hedgehogs  and  Foxes 

The  hedgehogs  have  one  large,  overriding  schema,  such  as  “axis  of  evil,”  with  which  they  parsimoniously 
analyze  world  affairs.  Foxes  are  less  wedded  to  one  overarching  world  view  and  instead  “improvise  ad  hoc 
solutions” 

Kozlowski 

2011 

Tetlock 

13.  Characteristics  of  the  intelligence  researcher:  reasoning  ability,  accuracy,  intellectual  honesty,  open- 
mindedness,  skepticism,  a  sense  of  detachment,  patience,  diligence,  perseverance,  and  imagination. 

Clauser  &Weir 

1976 

Unknown 

14.  High  Level  reasoning  ability.  Inductive  reasoning.  Intellectual  flexibility,  Writing  skill.  Memory,  Intellectual 
curiosity,  Deliberateness,  carefulness,  Interpersonal  skill,  Achievement  motivation,  Self-discipline, 

Perseverance 

Fischl  &  Gilbert 

1983 

Unknown 

15.  Research  also  demonstrated  the  following  personality  traits  most  often  seen  in  intelligence  analysts: 

"orientation  to  the  inner  world  of  ideas  rather  than  the  outer  world  of  things  and  people,  tendency  to  gather 
factual  information  through  senses  rather  than  inspiration,  proclivity  to  make  decisions  on  the  basis  of  logic 
rather  than  emotion,  and  an  eagerness  to  seek  closure  proactively  instead  of  leaving  possibilities  open. 

Ormond 

2002 

Krizan 

16.  The  personal  attributes  of  knowledge  workers  should  ideally  include  the  following: 

-  They  can  work  in  multiple  domains  simultaneously,  moving  in  and  out  of  them,  continuously  expanding  their 
knowledge,  capabilities,  perceptions,  capacities  and  networks. 

-  They  manage  knowledge  in  the  sense  of  recognising,  creating,  finding  and  moving  knowledge  that  is  valid, 
useful  and  applicable  to  the  issue  at  hand.  They  can  create  ideas,  solve  problems,  make  decisions  and  take 

Duvenage 

2010 

Bennet,  Alex 
and  David. 
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effective  action,  either  individually  or  as  a  group. 

-  They  have  foresight  to  sense  the  future  knowledge  needs  and  acquire  that  knowledge  to  handle  challenging 
problems  well  before  they  arise.  Their  understanding  of  systems  and  complexities  helps  them  to  identify 
possible  future  knowledge  needs. 

-  They  are  ongoing  learners  who  have  sound  discipline,  knowledge  and  a  broad  competency  that  spans  many 
dimensions.  This  implies  that  they  realise  that  they  cannot  be  experts  in  all  domains  and  are  therefore  willing 
to  forego  their  perspectives  and  beliefs  to  adopt  a  broader  understanding  of  an  issue  at  hand. 

-  They  are  convergent  thinkers  who  have  knowledge  of  systems,  complexities  and  critical  thinking  and  who 
can  use  different  approaches  and  techniques  to  better  understand  complex  issues. 

-  They  develop  and  nurture  their  relationship  networks  to  gain  knowledge  and  actions  in  new  environments. 

-  They  are  information  literate.  They  know  how  to  find,  evaluate  and  use  information  effectively  to  solve  a 
particular  problem  or  make  a  decision. 

-  In  knowledge  organisations,  where  change  happens  rapidly  and  the  creation  and  use  of  knowledge  to  gain  a 
competitive  advantage  is  paramount,  the  knowledge  workers  will  spend  more  time  learning,  thinking  and 
collaborating  and  less  time  applying  what  they  already  know. 

17.  The  effective  knowledge  workers  (or  intelligence  analysts)  are  capable  of  working  “in  multiple  domains 

simultaneously,  moving  in  and  out  of  those  domains  as  needed,  combining  the  physical,  mental,  the  intuitive, 
and  the  emotional  to  continuously  expand  their  knowledge,  capabilities,  capacity,  networks,  and  perceptions. 
They  are  convergent  thinkers  who  have  knowledge  of  systems,  complexity  and  critical  thinking  and  who  can 
use  different  approaches  and  techniques  to  better  understand  complex  issues.”  One  would  come  across  the 
concepts  “powers  of  abstraction”,  “conceptualisation”  and  “synthesis”  skills,  “out-of-the-box”  and 
“heterogeneous  thinking”  quite  often  in  knowledge-management  literature.  The  research  in  cognitive  and 
social  abilities  of  the  “global  worker”  informs  which  type  of  person  intelligence  agencies  should  recruit  or  co¬ 
opt;  probably  that  kind  of  person  who  might  not  pass  the  outdated  vetting  procedures  and  psychological  tests. 

Duvenage 

2010 

Bennet,  Alex 
and  David. 

18.  Strategic  human  resource  management  offers  an  objective,  scientific  approach  to  developing  the  best  possible 
workforce.  It  is  grounded  in  the  findings  that  individuals  differ  on  a  wide  range  of  psychological 
characteristics —  such  as  cognitive  ability,  personality,  and  values — that  predict  corresponding  differences  in 
educational  achievement,  job  performance,  and  career  success.  Some  of  these  characteristics  are  relatively 
stable,  such  as  cognitive  ability,  personality,  and  values,  while  others  are  more  malleable,  such  as  job 
knowledge,  job-specific  skills,  attitudes,  and  motivational  characteristics. 

Fingar 

2011 

Unknown 

19.  4. 1  Accurate  memory 

4.2  Flexibility 

4.3  Integrity  and  moral  courage 

4.4  Self-starter 

Derbentseva, 

McLellan, 

Mandel 

2010 

Interviews 
with  analysts 
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4.5  Thirst  for  knowledge 

4.6  Thoroughness 
20.  Insatiably  curious 

Self-motivated 
Fascinated  by  puzzles 
Exhibits  “AHA”  thinking 
Observes  voraciously 
Reads  voraciously 
Fruitfully  obsessed 
Takes  variable  perspectives 
Makes  creative  connections 
Playful 

Has  sense  of  humour 
Has  sense  of  wonder 
Concentrates  intensely 
Questions  convention 
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Derbentseva, 

McLellan, 

Mandel 

2010 

Moore  & 
Krizan 
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LIST  OF  ACRONYMS 


ACT 

American  College  Test 

AET 

Argument  Evaluation  Test 

AOT 

Actively  Open-Minded  Thinking  (also  AOMT) 

APA 

American  Philosophical  Association 

AT 

Ambiguity  Tolerance 

CCTDI 

California  Critical  Thinking  Dispositions  Inventory 

CCTST 

California  Critical  Thinking  Skills  Test 

CEC 

Connect-Extend-Challenge 

CPI 

California  Psychological  Inventory 

CRT 

Cognitive  Reflection  Test 

CT 

Critical  Thinking 

DMC 

Decision  Making  Competence 

EFA 

Exploratory  Factor  Analysis 

FAC 

Framework  for  Analytic  Cognition 

GPA 

Grade  Point  Average 

GRE 

Graduate  Record  Examination 

IC 

Intelligence  Community 

IQ 

Intelligence  quotient 

MBTI 

Meyers-Briggs  Type  Indicator 

MD 

Medical  Doctor 

NFC 

Need  for  Cognition  (also  as  NC) 

NFCC 

Need  for  Closure  Scale 

NFCS 

Need  for  Cognition  Scale 

NCLEX 

National  Council  Licensure  Examination 

SAT 

Scholastic  Aptitude  Test 

TDK 

Thinking  Dispositions  Questionnaire 

TSD-PI 

Trait  Self  Descriptive  Personality  Inventory 
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